From nobody Mon Feb 9 04:31:46 2026 Received: from mail-wr1-f51.google.com (mail-wr1-f51.google.com [209.85.221.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A8DFE14F70 for ; Sun, 18 Aug 2024 06:36:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963006; cv=none; b=AlLCWvfESMUk5VgGsre2MphwjWS3orOVk03wrqnLdPR5dDfsjQHwBEcVdWO9oYvLXFZnAyZbQPuLjGHGLkghSrCMx+WJyt89TsPIRTL1RdZTUQd628hlvazd6NHfLfcKmP3xquP9Du5Z2/+RxBUP37MYA9eJL1I/AiBl3SJd8FU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963006; c=relaxed/simple; bh=3vAJDoEpGiFg4mI/K3dJV6TDhU2SVBU3QlsfeYAyaSY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=SRx8VkhwDtraCQ4jU1QAs7qnLtFrfzMCCD9vUFmOPFkr7BEaU6E67o5FNNdkpAPpXfHPxTf9rcVY1K/rfYDEagUt6aWQzQmWOd9RWNqqFVz3B2zvDotK35vlUMCQmpTnOTZQK60PV+18E0X0KGHGqga8soXCt1bK21aK575qt/8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=yaRqC3H6; arc=none smtp.client-ip=209.85.221.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="yaRqC3H6" Received: by mail-wr1-f51.google.com with SMTP id ffacd0b85a97d-3718706cf8aso2064560f8f.3 for ; Sat, 17 Aug 2024 23:36:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963003; x=1724567803; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ll6PO6fq7jekV3yT3oRS1DwKakMdlEGxiDtEOFhGbwU=; b=yaRqC3H6BddAUcVAIIm5XKgPrKVM6Es6Rh9hwBvijOgOJ5DgscQDmRSs6NYazp1wxV 5Txwp9nUhJpjIB2tFbMGj7lxchQ23HnDXoUNKNeYK57HtXTLUBrfeNrFKYscD2eFTyzB GCczJoK1HEFxtiJ3byYGW/Hm0BbL4O0tczOs15r6QYwQi5X1NS6xcd2C4PYgoqT4dXg5 2P8DVlQ80pygGY8vMSYPv9w91DenJottsjPuWbHhBbu9CkoNQJtLhngonhwiTcfYy5eG vRDHc+eLSXbSFQ8AgUg0I8XIXYxs0U9CoMnMJBCh8tTs7oTATS+hWBdmM2pUxGWhKcQ0 nglg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963003; x=1724567803; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ll6PO6fq7jekV3yT3oRS1DwKakMdlEGxiDtEOFhGbwU=; b=bcDE5OB7/aO/w9pxu3/qMhsAfRvCiQim5moOUYyz1zyQCvJXoxagHlWIrFJZGKaZur fEzqLRsmXaoLCdEneTu7FO93JYSo+o09DsBykCr6uVyQJbNmx44k4D236lNrrKRrbZTc VeotixRL7BWbymnJTNPYS1CBTPiPQeYnDnld8PDKiKyeZ9SANVq+2fE9DM7RCDlpc12o TZWCn/lyUC6EQeoIK6jnubDkRroZKKQXGlqBN7VXvCVT7n8z1mHNAJrCD7O4Y8Foi9Wl jAVveTpHIUVdWBTBO6aZuGCbLKjoKctUT/kfzgJ0XnmYIENF2+mcG5uCwA7SfruW9TbR 2m7g== X-Forwarded-Encrypted: i=1; AJvYcCXKLtmrCdVbTSQkS1ShuWtrhIEgd5vNc5yB3KBnznMTDh8ArxZV78/re2TO361mBWisXSPxoWrltzrHRZYPKWv4LUNO6enaNelVhfA0 X-Gm-Message-State: AOJu0YxVtw9nFo5WkF1spQKruNy/HlkfFPeQf/wCtCaAENrNFI1xL4Af MyIoWxqBW6opLIK+LsxID6EC36RQS0u4EXzAtNkQ2scT2cS7VF8SNriEX6DSyfE= X-Google-Smtp-Source: AGHT+IEMupb/eaKm5/+VPG7gQL1BT1ElhtPyyJ9e/FdDd2FVBOCIDCIy/gU0Zn16LvTEMnl2YJ5n3Q== X-Received: by 2002:a5d:4148:0:b0:368:5ba0:622 with SMTP id ffacd0b85a97d-371946a4203mr4585432f8f.44.1723963002780; Sat, 17 Aug 2024 23:36:42 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-37189896de9sm7251293f8f.75.2024.08.17.23.36.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:36:42 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v5 01/13] riscv: Move cpufeature.h macros into their own header Date: Sun, 18 Aug 2024 08:35:26 +0200 Message-Id: <20240818063538.6651-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" asm/cmpxchg.h will soon need riscv_has_extension_unlikely() macros and then needs to include asm/cpufeature.h which introduces a lot of header circular dependencies. So move the riscv_has_extension_XXX() macros into their own header which prevents such circular dependencies by including a restricted number of headers. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/include/asm/cpufeature-macros.h | 66 ++++++++++++++++++++++ arch/riscv/include/asm/cpufeature.h | 61 ++------------------ 2 files changed, 70 insertions(+), 57 deletions(-) create mode 100644 arch/riscv/include/asm/cpufeature-macros.h diff --git a/arch/riscv/include/asm/cpufeature-macros.h b/arch/riscv/includ= e/asm/cpufeature-macros.h new file mode 100644 index 000000000000..a8103edbf51f --- /dev/null +++ b/arch/riscv/include/asm/cpufeature-macros.h @@ -0,0 +1,66 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright 2022-2024 Rivos, Inc + */ + +#ifndef _ASM_CPUFEATURE_MACROS_H +#define _ASM_CPUFEATURE_MACROS_H + +#include +#include + +#define STANDARD_EXT 0 + +bool __riscv_isa_extension_available(const unsigned long *isa_bitmap, unsi= gned int bit); +#define riscv_isa_extension_available(isa_bitmap, ext) \ + __riscv_isa_extension_available(isa_bitmap, RISCV_ISA_EXT_##ext) + +static __always_inline bool __riscv_has_extension_likely(const unsigned lo= ng vendor, + const unsigned long ext) +{ + asm goto(ALTERNATIVE("j %l[l_no]", "nop", %[vendor], %[ext], 1) + : + : [vendor] "i" (vendor), [ext] "i" (ext) + : + : l_no); + + return true; +l_no: + return false; +} + +static __always_inline bool __riscv_has_extension_unlikely(const unsigned = long vendor, + const unsigned long ext) +{ + asm goto(ALTERNATIVE("nop", "j %l[l_yes]", %[vendor], %[ext], 1) + : + : [vendor] "i" (vendor), [ext] "i" (ext) + : + : l_yes); + + return false; +l_yes: + return true; +} + +static __always_inline bool riscv_has_extension_unlikely(const unsigned lo= ng ext) +{ + compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_= MAX"); + + if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) + return __riscv_has_extension_unlikely(STANDARD_EXT, ext); + + return __riscv_isa_extension_available(NULL, ext); +} + +static __always_inline bool riscv_has_extension_likely(const unsigned long= ext) +{ + compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_= MAX"); + + if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) + return __riscv_has_extension_likely(STANDARD_EXT, ext); + + return __riscv_isa_extension_available(NULL, ext); +} + +#endif /* _ASM_CPUFEATURE_MACROS_H */ diff --git a/arch/riscv/include/asm/cpufeature.h b/arch/riscv/include/asm/c= pufeature.h index 45f9c1171a48..87ed88fc950d 100644 --- a/arch/riscv/include/asm/cpufeature.h +++ b/arch/riscv/include/asm/cpufeature.h @@ -8,9 +8,11 @@ =20 #include #include +#include +#include +#include #include -#include -#include +#include =20 /* * These are probed via a device_initcall(), via either the SBI or directly @@ -103,61 +105,6 @@ extern const size_t riscv_isa_ext_count; extern bool riscv_isa_fallback; =20 unsigned long riscv_isa_extension_base(const unsigned long *isa_bitmap); - -#define STANDARD_EXT 0 - -bool __riscv_isa_extension_available(const unsigned long *isa_bitmap, unsi= gned int bit); -#define riscv_isa_extension_available(isa_bitmap, ext) \ - __riscv_isa_extension_available(isa_bitmap, RISCV_ISA_EXT_##ext) - -static __always_inline bool __riscv_has_extension_likely(const unsigned lo= ng vendor, - const unsigned long ext) -{ - asm goto(ALTERNATIVE("j %l[l_no]", "nop", %[vendor], %[ext], 1) - : - : [vendor] "i" (vendor), [ext] "i" (ext) - : - : l_no); - - return true; -l_no: - return false; -} - -static __always_inline bool __riscv_has_extension_unlikely(const unsigned = long vendor, - const unsigned long ext) -{ - asm goto(ALTERNATIVE("nop", "j %l[l_yes]", %[vendor], %[ext], 1) - : - : [vendor] "i" (vendor), [ext] "i" (ext) - : - : l_yes); - - return false; -l_yes: - return true; -} - -static __always_inline bool riscv_has_extension_unlikely(const unsigned lo= ng ext) -{ - compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_= MAX"); - - if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) - return __riscv_has_extension_unlikely(STANDARD_EXT, ext); - - return __riscv_isa_extension_available(NULL, ext); -} - -static __always_inline bool riscv_has_extension_likely(const unsigned long= ext) -{ - compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_= MAX"); - - if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) - return __riscv_has_extension_likely(STANDARD_EXT, ext); - - return __riscv_isa_extension_available(NULL, ext); -} - static __always_inline bool riscv_cpu_has_extension_likely(int cpu, const = unsigned long ext) { compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_= MAX"); --=20 2.39.2 From nobody Mon Feb 9 04:31:46 2026 Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4314317557 for ; Sun, 18 Aug 2024 06:37:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963067; cv=none; b=EVMOokOUbCp6V096+aFGyHLo7IwgM46fcX6UJmJJDvJeL4Zp0Wc7RVbvTY7IqvZRzhtSy4p1xuyTvp48anAlo1Wt8+OskuMdYotCXzY/E/5f5+mMUc0X+E2jDa+8D6W/aKiVBky2fq8iAtYn7Vt5RWVvoRq8bGU9YyR3uwGyHmI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963067; c=relaxed/simple; bh=p5C0mPrYaEV0ZHTAtYTAehGo4qZczBizeN+6aCANZ+E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ugqMBt2YW7tRxMBSebkk20VFz7kABybXDgdi13mri6vBy1bgUIEdzXZr1vu1psaRm0nBoSykhSmCIPK1W1Bi+O+SFyH3iRhGOy78+tlZNcHSDhHsP7BbgZhn5C6TpTorzjw779/GtvZQiKMnBEh2XXaHqtzpxeg5t9IhGIBDKho= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=UdVKXF3A; arc=none smtp.client-ip=209.85.128.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="UdVKXF3A" Received: by mail-wm1-f47.google.com with SMTP id 5b1f17b1804b1-428178fc07eso23515625e9.3 for ; Sat, 17 Aug 2024 23:37:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963064; x=1724567864; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=baf+ls/kqppHRVa4XBeUzS5UcHJPfW6ACwqFNpH2vH8=; b=UdVKXF3Awl1w29iyh5jWcrKC181zC+3B0oOIr0O7IABxFn37cqqyDpvZPtuRatadYV 9xDNhr4TFm2E4OL+1BoUYfwbutLFTdf9Y2/CbW/pCFPp8iw8Bu7bB6KDr/c7TJ9fSU+W CYe0+nAurfkqxHegeIKFYIsuoAa0JZ6/LmKFo80PX/bDeBmnMPpXHX+71g71DGDipfZ6 KxHbEPdTT5hHlO3VvJdBczZqUD9Xpf8QCjgZ6JTa48VLwgAFOggO6p762ymNsgm7BW4S ONMedDD+buiolZmxEoe5AwegFmPRbFEWYYDG934z62Xr7K8ruin3iwAwCoagxyiQa7sE Ymaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963064; x=1724567864; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=baf+ls/kqppHRVa4XBeUzS5UcHJPfW6ACwqFNpH2vH8=; b=tR5GoR8Ajw3/cScMb8Jtgn33LW2FWWGtk/N2iRuXcjtGWp2DvmpyOPxnVSvekX91QX 16vpbwqObHkSkBDwtiwk8UlDTi4Uc50HxJuysSj6rGXbaCvVaRngZBV/hMJiO6EFJarw SMCW8Zz8bA8/4wmzhz4EcqNTE8cq+OJEniTQUFedSo96eZyA/Xb9L9x19tJGHUGJhAlW SJNmt+EbfY/g2BU4SjPQAqkWnt4DxfN0InSCneCyq7tntTgAoHWj1AKjTyGvwrHmnpFy f6waOwTOYlhfak3Z618eoJ1+QRzmGRxWjelJd3VAbphvnJYtL9f5vrVVTM+FbY/PFlmf i7RA== X-Forwarded-Encrypted: i=1; AJvYcCX5f3RvWKUAnyVCe36OYxNkAv5aFIFYVrvhktNUIUXTp4W5QK5p8YNM3VdLc6tcVmbp0p1Ev8fTiJaDNoDjfkJ+AJalycQve8L5jV3X X-Gm-Message-State: AOJu0YwLFoMsfoQeLzTLGn2UzMHY6MBoR3OSQZQVZRPUIi7Vbmo9/APe TaWnNQn0LjY1RbylLtG6ZFceFqlKryp3ys1SvoOfR1cuHeWrhKH98TRQTQXWdQw= X-Google-Smtp-Source: AGHT+IGOB8iW+hQOXjQLAOh7AsPotaVJZqzzaXjL4GlISP37ETZjWU8j89260z8kTNAIQ1PuXHkP4A== X-Received: by 2002:a05:600c:3b1f:b0:426:629f:1556 with SMTP id 5b1f17b1804b1-429ed7f9224mr53506085e9.31.1723963064325; Sat, 17 Aug 2024 23:37:44 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-429ded7c93esm119862495e9.41.2024.08.17.23.37.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:37:43 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v5 02/13] riscv: Do not fail to build on byte/halfword operations with Zawrs Date: Sun, 18 Aug 2024 08:35:27 +0200 Message-Id: <20240818063538.6651-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" riscv does not have lr instructions on byte and halfword but the qspinlock implementation actually uses such atomics provided by the Zabha extension, so those sizes are legitimate. Then instead of failing to build, just fallback to the !Zawrs path. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/include/asm/cmpxchg.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index ebbce134917c..ac1d7df898ef 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -245,6 +245,11 @@ static __always_inline void __cmpwait(volatile void *p= tr, : : : : no_zawrs); =20 switch (size) { + case 1: + fallthrough; + case 2: + /* RISC-V doesn't have lr instructions on byte and half-word. */ + goto no_zawrs; case 4: asm volatile( " lr.w %0, %1\n" --=20 2.39.2 From nobody Mon Feb 9 04:31:46 2026 Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D4A65DF51 for ; Sun, 18 Aug 2024 06:38:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963129; cv=none; b=cjBV+lmS9QphgCdYyh1dekRo1NBUhStMBQcmco47GnBDi9jyGf/MLzz1xfFtQtRecle7ojmkDkbL/zx8klnrXws441WzI2JG+QscQ0iGZRxjK8WBDd1QPeTpF/++lZ8d3forwTc3q+DFKy0Jk+AVJYrS7LB5kZdCQM5Az3oBYXg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963129; c=relaxed/simple; bh=gDF2VWVMCr0OWTZfcqqsgA0khsQjVyY2MClD81Q70wk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Cna9M794ZGcQWfhae4pn5SVBBVd5XWzboGycLZZlssfjvePXLGmKuquvxtjwu7u7akQrj3hHhuQOL/MLsaN2vxDmZxjZzRn7pUAyOqvTtgvodW+xYtnzs27RMNM0UQrIKPGzbTRO3WI6kogZwVSTT4AUHZrbaVmmtxSwRW2QgHE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=ciyggjcf; arc=none smtp.client-ip=209.85.128.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="ciyggjcf" Received: by mail-wm1-f48.google.com with SMTP id 5b1f17b1804b1-428fb103724so21881565e9.1 for ; Sat, 17 Aug 2024 23:38:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963126; x=1724567926; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dgpIYh7KuQW+iTbAJx3HXjQmOubS3QQ17d+o/pur+bQ=; b=ciyggjcfKkyG5WWx6+/WxNu5UaYeFgIcQdY1xWJp/Fruonsq2Y44MtGctFDXCIqUmV BEeSSXuLLw45MixLQow2FkXgDnq428PJM6BzJwmHcsXWUSaEmm95YVWGo7zbgJ7Mo/Bh K973bQNsvpT3Ko6RQtP3L6WrJyq5a+v3/UM2qXDrepFATglW3AJ0cvAqDHNc9pFcnceb rspFMVDQL1ziceD48Z7/PmOQgIoQbr/pPu/sfLW0X+dz+oCHNEll52j8QI0FCGLyIcUZ hQ1Jape/uTMpWlp8yfp8XhTbX6RHH+oE2GZbKckLydlW94cC7VHI7sJz8L85AaEkehwF 0POA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963126; x=1724567926; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dgpIYh7KuQW+iTbAJx3HXjQmOubS3QQ17d+o/pur+bQ=; b=AxiYct2iYoIfCIgO8ewX5Zb0FBobjhkfkIyWOXplhe6U8ZuUVSTs3VnQtpCBIrOIkT yIGTq4xa3RbjbQiFS+ztYbxkgdhkoq8NrCZ1e+z9vkBLNY3h9J1OIK6UHPzzP6Gqv8qO iGPgy00md8DfUvPDx5h3JKtWMSG6iAyoj31mPEzVaP0k7jnTnfkahVa39lR+JufaZrq5 gp/AM4Wu8Fl7WXDZx7OiqEzg7NPjdMLDbmZomn5UagDYYhba+Z4D+YbLK4EExrFmTz9P TrZSCHraVxUIpjF8XNa4eIH3gvu2vzCkU7AD5LP9ZUW4fTNkTTCxkxXeQXPRjOUuS69i SEiQ== X-Forwarded-Encrypted: i=1; AJvYcCVtO0X4tZ6XdrOxmqiSazpOQvv5C86z8rLfDSLj2Ng/dtZmZhbSB7enLuqAirfpBVgj0rEGYnjOR9gdt8If2LuVGWDEgs46QH9dO827 X-Gm-Message-State: AOJu0YzctTiAEvCvrkltVvX+ywBrPMfcDx6Yj1Cc+VbK/13t7L46mt7N XPI9Vo+H6cM8I0kX3ly/lxCOT8PlBNCjD9aJZoMOCwxEnGKJzGO945kztCVCI6w= X-Google-Smtp-Source: AGHT+IH8V6P8ozp0CwQCJM5ywpee/RSEJ1cgWdpG4qWioSCMxPEkZ3gtZHRrHCz0XJWkyE2olycTYw== X-Received: by 2002:adf:f648:0:b0:366:ee84:6a77 with SMTP id ffacd0b85a97d-37186bbd203mr7255006f8f.3.1723963125945; Sat, 17 Aug 2024 23:38:45 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-429eeadfafbsm62053035e9.47.2024.08.17.23.38.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:38:45 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v5 03/13] riscv: Implement cmpxchg32/64() using Zacas Date: Sun, 18 Aug 2024 08:35:28 +0200 Message-Id: <20240818063538.6651-4-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This adds runtime support for Zacas in cmpxchg operations. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/Kconfig | 16 +++++++++++ arch/riscv/Makefile | 3 ++ arch/riscv/include/asm/cmpxchg.h | 48 +++++++++++++++++++++----------- 3 files changed, 50 insertions(+), 17 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 0f3cd7c3a436..d955c64d50c2 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -613,6 +613,22 @@ config RISCV_ISA_ZAWRS use of these instructions in the kernel when the Zawrs extension is detected at boot. =20 +config TOOLCHAIN_HAS_ZACAS + bool + default y + depends on !64BIT || $(cc-option,-mabi=3Dlp64 -march=3Drv64ima_zacas) + depends on !32BIT || $(cc-option,-mabi=3Dilp32 -march=3Drv32ima_zacas) + depends on AS_HAS_OPTION_ARCH + +config RISCV_ISA_ZACAS + bool "Zacas extension support for atomic CAS" + depends on TOOLCHAIN_HAS_ZACAS + depends on RISCV_ALTERNATIVE + default y + help + Enable the use of the Zacas ISA-extension to implement kernel atomic + cmpxchg operations when it is detected at boot. + If you don't know what to do here, say Y. =20 config TOOLCHAIN_HAS_ZBB diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index 6fe682139d2e..f1788131d5fe 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -82,6 +82,9 @@ else riscv-march-$(CONFIG_TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI) :=3D $(riscv= -march-y)_zicsr_zifencei endif =20 +# Check if the toolchain supports Zacas +riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZACAS) :=3D $(riscv-march-y)_zacas + # Remove F,D,V from isa string for all. Keep extensions between "fd" and "= v" by # matching non-v and non-multi-letter extensions out with the filter ([^v_= ]*) KBUILD_CFLAGS +=3D -march=3D$(shell echo $(riscv-march-y) | sed -E 's/(rv3= 2ima|rv64ima)fd([^v_]*)v?/\1\2/') diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index ac1d7df898ef..39c1daf39f6a 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -12,6 +12,7 @@ #include #include #include +#include =20 #define __arch_xchg_masked(sc_sfx, prepend, append, r, p, n) \ ({ \ @@ -137,24 +138,37 @@ r =3D (__typeof__(*(p)))((__retx & __mask) >> __s); \ }) =20 -#define __arch_cmpxchg(lr_sfx, sc_sfx, prepend, append, r, p, co, o, n) \ +#define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n= ) \ ({ \ - register unsigned int __rc; \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS)) { \ + r =3D o; \ \ - __asm__ __volatile__ ( \ - prepend \ - "0: lr" lr_sfx " %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc" sc_sfx " %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - append \ - "1:\n" \ - : "=3D&r" (r), "=3D&r" (__rc), "+A" (*(p)) \ - : "rJ" (co o), "rJ" (n) \ - : "memory"); \ + __asm__ __volatile__ ( \ + prepend \ + " amocas" sc_cas_sfx " %0, %z2, %1\n" \ + append \ + : "+&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + } else { \ + register unsigned int __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr" lr_sfx " %0, %2\n" \ + " bne %0, %z3, 1f\n" \ + " sc" sc_cas_sfx " %1, %z4, %2\n" \ + " bnez %1, 0b\n" \ + append \ + "1:\n" \ + : "=3D&r" (r), "=3D&r" (__rc), "+A" (*(p)) \ + : "rJ" (co o), "rJ" (n) \ + : "memory"); \ + } \ }) =20 -#define _arch_cmpxchg(ptr, old, new, sc_sfx, prepend, append) \ +#define _arch_cmpxchg(ptr, old, new, sc_cas_sfx, prepend, append) \ ({ \ __typeof__(ptr) __ptr =3D (ptr); \ __typeof__(*(__ptr)) __old =3D (old); \ @@ -164,15 +178,15 @@ switch (sizeof(*__ptr)) { \ case 1: \ case 2: \ - __arch_cmpxchg_masked(sc_sfx, prepend, append, \ + __arch_cmpxchg_masked(sc_cas_sfx, prepend, append, \ __ret, __ptr, __old, __new); \ break; \ case 4: \ - __arch_cmpxchg(".w", ".w" sc_sfx, prepend, append, \ + __arch_cmpxchg(".w", ".w" sc_cas_sfx, prepend, append, \ __ret, __ptr, (long), __old, __new); \ break; \ case 8: \ - __arch_cmpxchg(".d", ".d" sc_sfx, prepend, append, \ + __arch_cmpxchg(".d", ".d" sc_cas_sfx, prepend, append, \ __ret, __ptr, /**/, __old, __new); \ break; \ default: \ --=20 2.39.2 From nobody Mon Feb 9 04:31:46 2026 Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5CA08168B1 for ; Sun, 18 Aug 2024 06:39:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963191; cv=none; b=t53j1JYTXX0NvPFouuYTT44fxT/kDsvPcYhkazKoT1yFzDmtsq+p3tcJ0LxLj0//eZcNxGnBuIiXg+rPpxuLIKAh6/GpmWdEkERXy9oZ4n6kmLSmlqz2jlfHtv6p1kkC411r2CXL1R5pd5d7KKJfL2Tz6jqxdwx80EU1U/SzZy0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963191; c=relaxed/simple; bh=tnccALh8+1g50zeQtOru1Sy59cZzidIrCrT5x9+SPLo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tqMYxENN6veE08OVtdYjy4GWNCGjlyaWB7rTlkTfhi4FIWCOoWt8G1ZJsPv+LnO8iRm7XmKTi1h693zwYaTOAss7IqWDlEUfhyac47I07tyo2AUjocLM/WDvcCh6J2K2EsArMkzNJ0lDOWKdUvZAZNhI+D099lVjbIdZ4jxLQ74= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=0kmnSv2S; arc=none smtp.client-ip=209.85.128.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="0kmnSv2S" Received: by mail-wm1-f44.google.com with SMTP id 5b1f17b1804b1-428e3129851so25485685e9.3 for ; Sat, 17 Aug 2024 23:39:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963188; x=1724567988; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5sYtVuKxVaNEjuRyviwIAJgcsawGj7aKXy8R1Bii4eU=; b=0kmnSv2SZcmYF2iOoSRbkaouWHjGCeErm5JXreyGbLty424rwqoLGX3pBvygilnniT IAmg6ClUZqGDF48BEkBxPSipnoSmKGXTt1ILJnZdqgeqdc7sC27etLnA7pMjd947O0FL 1Tan+08wLS9mif7Oj+y/hnvbjm5jShSJFsGzkk8WW9uKSX4TVPpPh8rZdBdYUnNWaH41 25o6/1g5BQv5hkTE/KBsIL1eFEXEQvYgS3yFjmtgyN4tXIHGG8ljcuhsrVlpu4GE6ieg iwM6FLTSZyPwdy3SCE1HAv6iM62FFpyCvrC6bi3Zuql7/z1JAi4DM6ggrqoEaNw3vHsr LPmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963188; x=1724567988; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5sYtVuKxVaNEjuRyviwIAJgcsawGj7aKXy8R1Bii4eU=; b=C69lH1rbW/IVx5Z1aIMQ3MOUPgz/1nX/MeUNpvHZ+c1grN5IhTQo4XkByV8LHGnmVv 70haNyszxc+22gen2X3tqyOUyZNpJdOMCbtEaG+eEr55tV1obs8ex17LHVT4eIuCFKny KAoUW0OGEtVOk1ym/JqlklWXXhX3GezeB1B/wgY1JjfJCvDNYRlwqQuj1COD7vZCRmGQ QcuubEIHgL7HCTHJ1rjpbYw9V83r77QcbaW5maI0LSMUTsoXtR/RdRgyA9p4IxirmH6x EX3JEI5x64Fi/BWzwpta1pe36zNZIHxjSnBUgWvEHwBE/HWGnr5smxGzQL9SDT4jfp/L K4Wg== X-Forwarded-Encrypted: i=1; AJvYcCVYUtho0SmREy8l2fPJvjVUafOYl3BVf5sM8XeRhuvCzdqOMnKCGuhfWVlhNWS5MsdvO+6yoyjJeQ3Yd+EXAhHxA7qUJC0w79p9ptqg X-Gm-Message-State: AOJu0YwcMmbw2xfl4iAKldLt8swv32ub9cOWPA85NdVSYfAdoMTbjRvm 5+5KvevcWQbhkvY6U7Q129Q3roY2U7e3lL+lO5sM5O4bXBWM1JfqJT359TZS0xo= X-Google-Smtp-Source: AGHT+IGGAMdr2P1EdP5GWl4akDsUdOuePbLIYOG470k8Ym7dIUmRU5AiPcQ2o3G7Nw7Sy6NLXGgTQw== X-Received: by 2002:a05:600c:3c9e:b0:427:98b4:624b with SMTP id 5b1f17b1804b1-429ed7e4314mr53197535e9.24.1723963187547; Sat, 17 Aug 2024 23:39:47 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-429ed658e9esm66744295e9.27.2024.08.17.23.39.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:39:47 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Conor Dooley Subject: [PATCH v5 04/13] dt-bindings: riscv: Add Zabha ISA extension description Date: Sun, 18 Aug 2024 08:35:29 +0200 Message-Id: <20240818063538.6651-5-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add description for the Zabha ISA extension which was ratified in April 2024. Signed-off-by: Alexandre Ghiti Reviewed-by: Guo Ren Acked-by: Conor Dooley Reviewed-by: Andrew Jones --- Documentation/devicetree/bindings/riscv/extensions.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Documentation/devicetree/bindings/riscv/extensions.yaml b/Docu= mentation/devicetree/bindings/riscv/extensions.yaml index a06dbc6b4928..a63578b95c4a 100644 --- a/Documentation/devicetree/bindings/riscv/extensions.yaml +++ b/Documentation/devicetree/bindings/riscv/extensions.yaml @@ -171,6 +171,12 @@ properties: memory types as ratified in the 20191213 version of the privil= eged ISA specification. =20 + - const: zabha + description: | + The Zabha extension for Byte and Halfword Atomic Memory Operat= ions + as ratified at commit 49f49c842ff9 ("Update to Rafified state"= ) of + riscv-zabha. + - const: zacas description: | The Zacas extension for Atomic Compare-and-Swap (CAS) instruct= ions --=20 2.39.2 From nobody Mon Feb 9 04:31:46 2026 Received: from mail-wr1-f51.google.com (mail-wr1-f51.google.com [209.85.221.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DCA59DDD2 for ; Sun, 18 Aug 2024 06:40:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963254; cv=none; b=D4Z6l0ZijeJoGyuX+qAXNtUTMR/3dF+6PtvrVqy+GS9HVlQMEZBVWRcIhYwjjNO1PMg9q6PAx7GhesWRk15wp0ruau6YXfyPm+5BfKNG5P2wCVR/hQg8ZNtSqjXeQZBJFcIFLjzN7HeaW819Dye5GPx3jaDebQa09DG4GWVGs64= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963254; c=relaxed/simple; bh=d146KF2PKTmU6liiBvLIfIT43KxtPmJKXqkUFUiewAU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=XPO917kh0dYuGzt0zpveUVbHY2DUseqcDRVT4Wgso/cLFZo7cPpufgW6p4TTAnImCvplQQ6j6lE4nH01vpbfcJen5TD0z+48Tp4oSP3mAxwUF8yHn87XopUjMLrgbF7mkTaJm+KtQFckSOfXR47h3d01NDu/lQfbTiwIC/RKqXA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=w7FGP3pO; arc=none smtp.client-ip=209.85.221.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="w7FGP3pO" Received: by mail-wr1-f51.google.com with SMTP id ffacd0b85a97d-37189d8e637so1903104f8f.3 for ; Sat, 17 Aug 2024 23:40:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963249; x=1724568049; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/ZGnwjJ9zA50deMtasLoWGJlTqCTlw/AR/rhuAE8IZk=; b=w7FGP3pOgwZuG4ZS20G8DlEeLVtI6PMAXwNPp0g7XnKaoC3pb+z0P8jcgaEYx3ueSU f+S1vdz9MwcsBVZ8SbifPXHW4QLxFp4wSV3NqMN3qxuFvUJJpnEhU868KYU9ZIviOUOl roxa3MzpI3Q9L7S/sn19i0A4wZRvk0A6T+qBB1yCmhZXKnW/5sc88s/ZpLPb0sTH/C7Q HzBS6n13t8kli7vqfgxPHTyH59M3PAQbnjf3Kv56vaq2DMduR9Uiki9ch0il259tyFaT +jwxRBxbAS6mTGQ7gQqLDBTk+sm5dw5OH0jOnQFZiMvfMYolTG0WGKzmB2zhjHAD4PvI dW/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963249; x=1724568049; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/ZGnwjJ9zA50deMtasLoWGJlTqCTlw/AR/rhuAE8IZk=; b=WetQHFUC8XhqB6LWotUBGBuZlQRWryYRcYGNdszdPBr7BSeGoduDhLpsV1CplGCNQR X32FyKLGwVpdCswVx/R+fGN0Yy5KlDymnOQ5LwbovSNBo7WEh49+hyp4hu1PQEAajqBz Oramekn8deBhDxXhyY8Jjuu2cyHeytAvze2uniM7Fh49DDq/tdwpM0mx0NnGepsiowqO CMUBETgUYiP/xcCI7VNoKA6UTsd/7UapCCjsvH3iwOvVFdZkanWpwjNtaZ/ZqethsF7B lEw4Y7golgog0kV2Sa40UnsiphjHLYtzc7XsjE4wSq0Pp5koPpkq2+ybBOunffZVeG4d fGjA== X-Forwarded-Encrypted: i=1; AJvYcCWlqplVlMwuRCZ4791RQZj40C1/dnT/Swqd/w9ZsEmz4kCiLwo2kRl9yL80UH4ZRYKRm8EZKGXRu3k9sKD+71aWfDBd9eS869NXSPLH X-Gm-Message-State: AOJu0YztQRpadNyMyJm8KMI3EG29dH5c7ejJcZScVZx2fQtH/DvXTSJ4 iLoUk99vgOkvLiK5FiqiCo+hpmULWVZIcnDkBmXkELsCXRkVrhGfaBk+ERCdRv8= X-Google-Smtp-Source: AGHT+IH5q0bFu/E9vYl6O9ntKbWhfQ/kqCOKwUu/eU0CrpOWwaixsLysU9gopA/iIxfwBIo3S9Ykcw== X-Received: by 2002:a5d:53c5:0:b0:371:7d84:9bef with SMTP id ffacd0b85a97d-37194651551mr4876064f8f.28.1723963249067; Sat, 17 Aug 2024 23:40:49 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-371898aabdesm7286232f8f.99.2024.08.17.23.40.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:40:48 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v5 05/13] riscv: Implement cmpxchg8/16() using Zabha Date: Sun, 18 Aug 2024 08:35:30 +0200 Message-Id: <20240818063538.6651-6-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This adds runtime support for Zabha in cmpxchg8/16() operations. Note that in the absence of Zacas support in the toolchain, CAS instructions from Zabha won't be used. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/Kconfig | 18 ++++++++ arch/riscv/Makefile | 3 ++ arch/riscv/include/asm/cmpxchg.h | 78 ++++++++++++++++++++------------ arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/kernel/cpufeature.c | 1 + 5 files changed, 72 insertions(+), 29 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index d955c64d50c2..212ec2aab389 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -613,6 +613,24 @@ config RISCV_ISA_ZAWRS use of these instructions in the kernel when the Zawrs extension is detected at boot. =20 +config TOOLCHAIN_HAS_ZABHA + bool + default y + depends on !64BIT || $(cc-option,-mabi=3Dlp64 -march=3Drv64ima_zabha) + depends on !32BIT || $(cc-option,-mabi=3Dilp32 -march=3Drv32ima_zabha) + depends on AS_HAS_OPTION_ARCH + +config RISCV_ISA_ZABHA + bool "Zabha extension support for atomic byte/halfword operations" + depends on TOOLCHAIN_HAS_ZABHA + depends on RISCV_ALTERNATIVE + default y + help + Enable the use of the Zabha ISA-extension to implement kernel + byte/halfword atomic memory operations when it is detected at boot. + + If you don't know what to do here, say Y. + config TOOLCHAIN_HAS_ZACAS bool default y diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index f1788131d5fe..f6dc5ba7c526 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -85,6 +85,9 @@ endif # Check if the toolchain supports Zacas riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZACAS) :=3D $(riscv-march-y)_zacas =20 +# Check if the toolchain supports Zabha +riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZABHA) :=3D $(riscv-march-y)_zabha + # Remove F,D,V from isa string for all. Keep extensions between "fd" and "= v" by # matching non-v and non-multi-letter extensions out with the filter ([^v_= ]*) KBUILD_CFLAGS +=3D -march=3D$(shell echo $(riscv-march-y) | sed -E 's/(rv3= 2ima|rv64ima)fd([^v_]*)v?/\1\2/') diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index 39c1daf39f6a..1f4cd12e4664 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -108,34 +108,49 @@ * indicated by comparing RETURN with OLD. */ =20 -#define __arch_cmpxchg_masked(sc_sfx, prepend, append, r, p, o, n) \ -({ \ - u32 *__ptr32b =3D (u32 *)((ulong)(p) & ~0x3); \ - ulong __s =3D ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ - ulong __mask =3D GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ - << __s; \ - ulong __newx =3D (ulong)(n) << __s; \ - ulong __oldx =3D (ulong)(o) << __s; \ - ulong __retx; \ - ulong __rc; \ - \ - __asm__ __volatile__ ( \ - prepend \ - "0: lr.w %0, %2\n" \ - " and %1, %0, %z5\n" \ - " bne %1, %z3, 1f\n" \ - " and %1, %0, %z6\n" \ - " or %1, %1, %z4\n" \ - " sc.w" sc_sfx " %1, %1, %2\n" \ - " bnez %1, 0b\n" \ - append \ - "1:\n" \ - : "=3D&r" (__retx), "=3D&r" (__rc), "+A" (*(__ptr32b)) \ - : "rJ" ((long)__oldx), "rJ" (__newx), \ - "rJ" (__mask), "rJ" (~__mask) \ - : "memory"); \ - \ - r =3D (__typeof__(*(p)))((__retx & __mask) >> __s); \ +#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, prepend, append, r, p, o, n= ) \ +({ \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && \ + IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZABHA) && \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS)) { \ + r =3D o; \ + \ + __asm__ __volatile__ ( \ + prepend \ + " amocas" cas_sfx " %0, %z2, %1\n" \ + append \ + : "+&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + } else { \ + u32 *__ptr32b =3D (u32 *)((ulong)(p) & ~0x3); \ + ulong __s =3D ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ + ulong __mask =3D GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ + << __s; \ + ulong __newx =3D (ulong)(n) << __s; \ + ulong __oldx =3D (ulong)(o) << __s; \ + ulong __retx; \ + ulong __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr.w %0, %2\n" \ + " and %1, %0, %z5\n" \ + " bne %1, %z3, 1f\n" \ + " and %1, %0, %z6\n" \ + " or %1, %1, %z4\n" \ + " sc.w" sc_sfx " %1, %1, %2\n" \ + " bnez %1, 0b\n" \ + append \ + "1:\n" \ + : "=3D&r" (__retx), "=3D&r" (__rc), "+A" (*(__ptr32b)) \ + : "rJ" ((long)__oldx), "rJ" (__newx), \ + "rJ" (__mask), "rJ" (~__mask) \ + : "memory"); \ + \ + r =3D (__typeof__(*(p)))((__retx & __mask) >> __s); \ + } \ }) =20 #define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n= ) \ @@ -177,8 +192,13 @@ \ switch (sizeof(*__ptr)) { \ case 1: \ + __arch_cmpxchg_masked(sc_cas_sfx, ".b" sc_cas_sfx, \ + prepend, append, \ + __ret, __ptr, __old, __new); \ + break; \ case 2: \ - __arch_cmpxchg_masked(sc_cas_sfx, prepend, append, \ + __arch_cmpxchg_masked(sc_cas_sfx, ".h" sc_cas_sfx, \ + prepend, append, \ __ret, __ptr, __old, __new); \ break; \ case 4: \ diff --git a/arch/riscv/include/asm/hwcap.h b/arch/riscv/include/asm/hwcap.h index 5a0bd27fd11a..f5d53251c947 100644 --- a/arch/riscv/include/asm/hwcap.h +++ b/arch/riscv/include/asm/hwcap.h @@ -92,6 +92,7 @@ #define RISCV_ISA_EXT_ZCF 83 #define RISCV_ISA_EXT_ZCMOP 84 #define RISCV_ISA_EXT_ZAWRS 85 +#define RISCV_ISA_EXT_ZABHA 86 =20 #define RISCV_ISA_EXT_XLINUXENVCFG 127 =20 diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index b427188b28fc..67ebcc4c9424 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -322,6 +322,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] =3D { __RISCV_ISA_EXT_DATA(zihintpause, RISCV_ISA_EXT_ZIHINTPAUSE), __RISCV_ISA_EXT_DATA(zihpm, RISCV_ISA_EXT_ZIHPM), __RISCV_ISA_EXT_DATA(zimop, RISCV_ISA_EXT_ZIMOP), + __RISCV_ISA_EXT_DATA(zabha, RISCV_ISA_EXT_ZABHA), __RISCV_ISA_EXT_DATA(zacas, RISCV_ISA_EXT_ZACAS), __RISCV_ISA_EXT_DATA(zawrs, RISCV_ISA_EXT_ZAWRS), __RISCV_ISA_EXT_DATA(zfa, RISCV_ISA_EXT_ZFA), --=20 2.39.2 From nobody Mon Feb 9 04:31:46 2026 Received: from mail-wr1-f50.google.com (mail-wr1-f50.google.com [209.85.221.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 57D8A14285 for ; Sun, 18 Aug 2024 06:41:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963314; cv=none; b=NuVSZJnBcbnsXWeJ6L0vwRSkFfDoJGrBuE3zwVkrChBiuZi5kyQb+LLbjFO0Ci+pmYYeA7NjXEvAf/jhacoyu7gdJO4EU+erajmow29dBca6YCQxrCKHRjOMRuaWt1kFudew6DLB7hS3ooV56NhdWuRoMFetvsuO6ASDvIFw1k0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963314; c=relaxed/simple; bh=49XBNMxYmaKb1kOfY1Da4EKIlSdrTdQzrpxHv533ook=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=mm67QM6TjW5AFw5RHyXMVitvfZopXMf2YcYYhu3tJuLhge9D6lmqVL9Uw0EB4KxnfUnME25vLQEHcWUhOQEdw3Evubz/pgFMiFaNlIEVEfCzIeeDLdNUyJDC6VUCptXPWx5HcG4IJqJ7B2gX+5h8s8QnFymrOl+udxLcX9WDohc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=CoW2ZonT; arc=none smtp.client-ip=209.85.221.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="CoW2ZonT" Received: by mail-wr1-f50.google.com with SMTP id ffacd0b85a97d-37196229343so1256860f8f.0 for ; Sat, 17 Aug 2024 23:41:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963311; x=1724568111; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EzrQ84GOc79Cvd4pehyYMEiToRaVIXWpNcTPRgBQZOo=; b=CoW2ZonTZKGkYgSDcskr/DbpxDb5Un6LTrw8TLUVLPruyYhtYsPHi8+L8Y8TpSeJUq latWMBkPnFPeWbltwfkNifYIbFx8LYeBz0MLLCkkJM3n5ZChDnU3rWRIvK415/x+3vvD 4F0MoBhRXlArSjkLL0p9A7W/UxXvN4MnYIQ6LPJo+Ljs1GJfex8XgWwrJXTvlQvztn3J AzaIO3LHy8uYevxxBjiEr3JiHTUfrRhrtzS58OLRzBqUp4Qmm9nprVhUbkhrhqBsDs1A RbRonYGDegZoF9oNy09lwq8g54scglDhHo2fOBb+K1ovMhQXFmuI03IafdRDox8QCjKe SO3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963311; x=1724568111; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EzrQ84GOc79Cvd4pehyYMEiToRaVIXWpNcTPRgBQZOo=; b=gfJuUK2Kl6g1oIOW17w6YhOEHd0jf+uHMVOUzkaViHoYycXoGqUoof0GTzAzO2MxPC c08aG2m/2tPpwM1/qOHRlZBZYzX4CGfOfcKz1KrFlSKNtttQEhnJ/5HBv3PMOMLov3in YOvDVJHNih8TthjIEJdbvKeru2gwaaSIv1YwaVlgj8puYIeh1UhNY0m9TVcxZw6U4hNm 8KxeOTmyy+tj3+wBShGNCjMm55L9/3mrFofmjMrUdg+GxyZuwNJteHrpCQtAoaBRyLVU dbf8sT/38aajIAYKSePL0bXMapdv5A/8X6Cw4lboRCQwRHfoSxbulOyXy3iicV9bl8pb 2DyQ== X-Forwarded-Encrypted: i=1; AJvYcCU4i7bu5/90yswIZy46CD1H3OCiRwvct+/Em74M18nXRintXeWOtBV/tykBeJXkmLttLug1lkevfLaeD3btyujv0mQWUSRHRqWvop1m X-Gm-Message-State: AOJu0YycKAsCIzozP8qprKOM/AQlGjqD6lFS6SV5GMD5vubf18gK2+67 dVqNvK3D6SHpKNl4jAuA8+Jnz69cZq0VHCa5rK4uAb63SXozn4M47pSQRWLYViw= X-Google-Smtp-Source: AGHT+IH7jXMI8vQlmJu9PzoH6e8IpPUsf9yyX42mqd9sjDsguutH4AFv/sh4mfXo/xTrWqv9pY4RtQ== X-Received: by 2002:adf:a1ce:0:b0:368:7fbc:4062 with SMTP id ffacd0b85a97d-371946531e9mr4631837f8f.33.1723963310553; Sat, 17 Aug 2024 23:41:50 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-37189849606sm7362530f8f.32.2024.08.17.23.41.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:41:50 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrea Parri Subject: [PATCH v5 06/13] riscv: Improve zacas fully-ordered cmpxchg() Date: Sun, 18 Aug 2024 08:35:31 +0200 Message-Id: <20240818063538.6651-7-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The current fully-ordered cmpxchgXX() implementation results in: amocas.X.rl a5,a4,(s1) fence rw,rw This provides enough sync but we can actually use the following better mapping instead: amocas.X.aqrl a5,a4,(s1) Suggested-by: Andrea Parri Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/include/asm/cmpxchg.h | 92 ++++++++++++++++++++++---------- 1 file changed, 64 insertions(+), 28 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index 1f4cd12e4664..5b2f95f7f310 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -107,8 +107,10 @@ * store NEW in MEM. Return the initial value in MEM. Success is * indicated by comparing RETURN with OLD. */ - -#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, prepend, append, r, p, o, n= ) \ +#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + r, p, o, n) \ ({ \ if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && \ IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \ @@ -117,9 +119,9 @@ r =3D o; \ \ __asm__ __volatile__ ( \ - prepend \ + cas_prepend \ " amocas" cas_sfx " %0, %z2, %1\n" \ - append \ + cas_append \ : "+&r" (r), "+A" (*(p)) \ : "rJ" (n) \ : "memory"); \ @@ -134,7 +136,7 @@ ulong __rc; \ \ __asm__ __volatile__ ( \ - prepend \ + sc_prepend \ "0: lr.w %0, %2\n" \ " and %1, %0, %z5\n" \ " bne %1, %z3, 1f\n" \ @@ -142,7 +144,7 @@ " or %1, %1, %z4\n" \ " sc.w" sc_sfx " %1, %1, %2\n" \ " bnez %1, 0b\n" \ - append \ + sc_append \ "1:\n" \ : "=3D&r" (__retx), "=3D&r" (__rc), "+A" (*(__ptr32b)) \ : "rJ" ((long)__oldx), "rJ" (__newx), \ @@ -153,16 +155,19 @@ } \ }) =20 -#define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n= ) \ +#define __arch_cmpxchg(lr_sfx, sc_sfx, cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + r, p, co, o, n) \ ({ \ if (IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \ riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS)) { \ r =3D o; \ \ __asm__ __volatile__ ( \ - prepend \ - " amocas" sc_cas_sfx " %0, %z2, %1\n" \ - append \ + cas_prepend \ + " amocas" cas_sfx " %0, %z2, %1\n" \ + cas_append \ : "+&r" (r), "+A" (*(p)) \ : "rJ" (n) \ : "memory"); \ @@ -170,12 +175,12 @@ register unsigned int __rc; \ \ __asm__ __volatile__ ( \ - prepend \ + sc_prepend \ "0: lr" lr_sfx " %0, %2\n" \ " bne %0, %z3, 1f\n" \ - " sc" sc_cas_sfx " %1, %z4, %2\n" \ + " sc" sc_sfx " %1, %z4, %2\n" \ " bnez %1, 0b\n" \ - append \ + sc_append \ "1:\n" \ : "=3D&r" (r), "=3D&r" (__rc), "+A" (*(p)) \ : "rJ" (co o), "rJ" (n) \ @@ -183,7 +188,9 @@ } \ }) =20 -#define _arch_cmpxchg(ptr, old, new, sc_cas_sfx, prepend, append) \ +#define _arch_cmpxchg(ptr, old, new, sc_sfx, cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append) \ ({ \ __typeof__(ptr) __ptr =3D (ptr); \ __typeof__(*(__ptr)) __old =3D (old); \ @@ -192,22 +199,28 @@ \ switch (sizeof(*__ptr)) { \ case 1: \ - __arch_cmpxchg_masked(sc_cas_sfx, ".b" sc_cas_sfx, \ - prepend, append, \ - __ret, __ptr, __old, __new); \ + __arch_cmpxchg_masked(sc_sfx, ".b" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, __old, __new); \ break; \ case 2: \ - __arch_cmpxchg_masked(sc_cas_sfx, ".h" sc_cas_sfx, \ - prepend, append, \ - __ret, __ptr, __old, __new); \ + __arch_cmpxchg_masked(sc_sfx, ".h" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, __old, __new); \ break; \ case 4: \ - __arch_cmpxchg(".w", ".w" sc_cas_sfx, prepend, append, \ - __ret, __ptr, (long), __old, __new); \ + __arch_cmpxchg(".w", ".w" sc_sfx, ".w" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, (long), __old, __new); \ break; \ case 8: \ - __arch_cmpxchg(".d", ".d" sc_cas_sfx, prepend, append, \ - __ret, __ptr, /**/, __old, __new); \ + __arch_cmpxchg(".d", ".d" sc_sfx, ".d" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, /**/, __old, __new); \ break; \ default: \ BUILD_BUG(); \ @@ -215,17 +228,40 @@ (__typeof__(*(__ptr)))__ret; \ }) =20 +/* + * Those macros are there only to make the arch_cmpxchg_XXX() macros + * more readable. + */ +#define SC_SFX(x) x +#define CAS_SFX(x) x +#define SC_PREPEND(x) x +#define SC_APPEND(x) x +#define CAS_PREPEND(x) x +#define CAS_APPEND(x) x + #define arch_cmpxchg_relaxed(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", "", "") + _arch_cmpxchg((ptr), (o), (n), \ + SC_SFX(""), CAS_SFX(""), \ + SC_PREPEND(""), SC_APPEND(""), \ + CAS_PREPEND(""), CAS_APPEND("")) =20 #define arch_cmpxchg_acquire(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", "", RISCV_ACQUIRE_BARRIER) + _arch_cmpxchg((ptr), (o), (n), \ + SC_SFX(""), CAS_SFX(""), \ + SC_PREPEND(""), SC_APPEND(RISCV_ACQUIRE_BARRIER), \ + CAS_PREPEND(""), CAS_APPEND(RISCV_ACQUIRE_BARRIER)) =20 #define arch_cmpxchg_release(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", RISCV_RELEASE_BARRIER, "") + _arch_cmpxchg((ptr), (o), (n), \ + SC_SFX(""), CAS_SFX(""), \ + SC_PREPEND(RISCV_RELEASE_BARRIER), SC_APPEND(""), \ + CAS_PREPEND(RISCV_RELEASE_BARRIER), CAS_APPEND("")) =20 #define arch_cmpxchg(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), ".rl", "", " fence rw, rw\n") + _arch_cmpxchg((ptr), (o), (n), \ + SC_SFX(".rl"), CAS_SFX(".aqrl"), \ + SC_PREPEND(""), SC_APPEND(RISCV_FULL_BARRIER), \ + CAS_PREPEND(""), CAS_APPEND("")) =20 #define arch_cmpxchg_local(ptr, o, n) \ arch_cmpxchg_relaxed((ptr), (o), (n)) --=20 2.39.2 From nobody Mon Feb 9 04:31:46 2026 Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0DF9D14A90 for ; Sun, 18 Aug 2024 06:42:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963375; cv=none; b=bL+hpjyVnu0BOZBfHDzs+YuIQnsqEBJgFutZ/MJUkJArkPELSCtsQYoT6DzG5+hrTE+KB50I4k+xxXwDc2ooRm3WrY1Ap61OV6FIlXVkmoZXFe8GFo5JG9DCucPkF/KC1y0hHtW3bnSo7D1SdlJU5g8Pnbr6BRHaQGZ9LfI3we4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963375; c=relaxed/simple; bh=sRPSYdpr70kDnUmWl16TfbREyKeQnnlWJPh7z2q//BU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rqUOV9QYmxjLO8QBzkwbYYM20V5WA1K1Ldy7W/jdrwqqQTwvstB8ShgrNcWfud2SM5MNoQe1ceMyN1FPB5QVGK+8uQM8RgwgBhqoePArr3p8Z5kcSFnokVTE//aaQXzvvl/tbLOveKI+e7HUCmxuGjbTLG8DJVcuxiY4efCAkU0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=tz5D02/u; arc=none smtp.client-ip=209.85.128.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="tz5D02/u" Received: by mail-wm1-f47.google.com with SMTP id 5b1f17b1804b1-4280bca3960so27868335e9.3 for ; Sat, 17 Aug 2024 23:42:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963372; x=1724568172; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=quUzKJl3l9W/iH7huB/Ox/rGh7MBHVzUurNoeTrSXJY=; b=tz5D02/uCgrRxmvxomjzVDYx5pZEu+KZK8Q3PJq87Jd+kTRif9MReBc5/meIvYDoCb WtOjFxY2EJe7lNubPfyqINmGbIu/ZEpeKF6i9T1YOv+AjBBpkqxbUJLL9E6Y3nbuMLtZ GPZl+pL8dezAeSSnsRURZf6BqXuplzsHTenx/PJlqAnWm4+FKggrhE4CZxAU3pW7kE8y Kb/8EOcCqa4o59DDuXFQM/vARh94eM/OXo69Pe37+hP88jKIxfz2tLk/gWuqsGvDy8VK 4PC4FyLrvpfBaXsL//WY3nDXS/k6cpBO/bRne6UL5vTN1AgHXLZzRf6TSSA+J7GJ9cN0 6TxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963372; x=1724568172; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=quUzKJl3l9W/iH7huB/Ox/rGh7MBHVzUurNoeTrSXJY=; b=o5RHmmR+UaCDfrML42tFGQ+SeIS9gZR2xYk/qr617rBsVgPpFSIVEUtJUXzLFeEHNZ p8gULhG9WnFW6zQ4XufQjTIgvwM2SJXfByIUBV14WRdfF74r5zrj54mZUe7/1N53kGeE t7Q9Jn2TKI2MYqQxVY81L5VhEcK7+2VHFu4e0FqL/gjOtqvDNqu46DvdSUuQ37QXoL3Q 2Zqm3DTtp8QnWWJA9fj+GJLxk8OLoDYibblw+VLR3G79fky6bBihQ6YIH6LpQP0KB/xg 7wMyT3SuglYg4O5D51CIMbM/m+6X2abb9jueE96KHgHbVXQZqQbMh2Z9t+FYgBAXEJwj lYrQ== X-Forwarded-Encrypted: i=1; AJvYcCW7QwNQ73+1afGFRl+SUOO4esFALcg1zghs5cEHl59bMU94vOzm98oI2AqaxA9BeE+BeymGCKueAGaKtIdb3a/eMY2JDo33rnHc/hX3 X-Gm-Message-State: AOJu0YyC0OS64Ota66yp/yo1AVIah4VcczlfANl0KqPN9M4FEcCz/tSE /Is5r5c3LBPXuS0Rx9DzG0+3KmXbK564/43q+wGhcNvy82R3+ZeiJV2klpihvMg= X-Google-Smtp-Source: AGHT+IEe0ieY97eTK1clBc3HGi9QZxo6cf+OA5t5L0JQLmZOTEmF4/dAe6bc3mWD2TnCj1Dfw0jsiA== X-Received: by 2002:a05:600c:3509:b0:426:6b92:387d with SMTP id 5b1f17b1804b1-429ed7b8860mr53677765e9.21.1723963372179; Sat, 17 Aug 2024 23:42:52 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-429ed7945b8sm66605265e9.44.2024.08.17.23.42.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:42:51 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v5 07/13] riscv: Implement arch_cmpxchg128() using Zacas Date: Sun, 18 Aug 2024 08:35:32 +0200 Message-Id: <20240818063538.6651-8-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that Zacas is supported in the kernel, let's use the double word atomic version of amocas to improve the SLUB allocator. Note that we have to select fixed registers, otherwise gcc fails to pick even registers and then produces a reserved encoding which fails to assemble. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/cmpxchg.h | 38 ++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 212ec2aab389..ef55ab94027e 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -111,6 +111,7 @@ config RISCV select GENERIC_VDSO_TIME_NS if HAVE_GENERIC_VDSO select HARDIRQS_SW_RESEND select HAS_IOPORT if MMU + select HAVE_ALIGNED_STRUCT_PAGE select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMALLOC if HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_HUGE_VMAP if MMU && 64BIT diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index 5b2f95f7f310..05ba8a8e2ef5 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -296,6 +296,44 @@ arch_cmpxchg_release((ptr), (o), (n)); \ }) =20 +#if defined(CONFIG_64BIT) && defined(CONFIG_RISCV_ISA_ZACAS) + +#define system_has_cmpxchg128() riscv_has_extension_unlikely(RISCV_= ISA_EXT_ZACAS) + +union __u128_halves { + u128 full; + struct { + u64 low, high; + }; +}; + +#define __arch_cmpxchg128(p, o, n, cas_sfx) \ +({ \ + __typeof__(*(p)) __o =3D (o); = \ + union __u128_halves __hn =3D { .full =3D (n) }; \ + union __u128_halves __ho =3D { .full =3D (__o) }; \ + register unsigned long t1 asm ("t1") =3D __hn.low; \ + register unsigned long t2 asm ("t2") =3D __hn.high; \ + register unsigned long t3 asm ("t3") =3D __ho.low; \ + register unsigned long t4 asm ("t4") =3D __ho.high; \ + \ + __asm__ __volatile__ ( \ + " amocas.q" cas_sfx " %0, %z3, %2" \ + : "+&r" (t3), "+&r" (t4), "+A" (*(p)) \ + : "rJ" (t1), "rJ" (t2) \ + : "memory"); \ + \ + ((u128)t4 << 64) | t3; \ +}) + +#define arch_cmpxchg128(ptr, o, n) \ + __arch_cmpxchg128((ptr), (o), (n), ".aqrl") + +#define arch_cmpxchg128_local(ptr, o, n) \ + __arch_cmpxchg128((ptr), (o), (n), "") + +#endif /* CONFIG_64BIT && CONFIG_RISCV_ISA_ZACAS */ + #ifdef CONFIG_RISCV_ISA_ZAWRS /* * Despite wrs.nto being "WRS-with-no-timeout", in the absence of changes = to --=20 2.39.2 From nobody Mon Feb 9 04:31:46 2026 Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BB05F11CBD for ; Sun, 18 Aug 2024 06:43:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963437; cv=none; b=jHCxQiHbmunsgCbWtHUcNGsChr3jKWtuHTYEZP1O4lw/Y67Q+kmd2K8+s2nNWR86bPh2wcAY70EKAo9e8pjWILwEuAriuQLcmzn+H2ji5SQ0dp0M8Tj6qVDYqBdiJbNQpv6CBB9qdwLoMTgdOSA4r6DJJOyKjGzc2mafbhuuHFA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963437; c=relaxed/simple; bh=/AAo3eBZLqu9YReB96/A0olqzAGnxvHLZxg6b3in/iI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=V+84Ro8QyK6p10XAUw0gtmgeJLe60n4Nf5cfClPZvxN/+4hWnEZfsjB9eVd4WAvjDm2UG+BdRoDmprz5kXC7uyP9wXS1RY6E6iFCbRMqm11lirpm+eaXDShLZvsoaXu2B3grnYai4dG38ttGyz+ILWIJIcyMALfhGlIrzq6UhcU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=UVwpk8KY; arc=none smtp.client-ip=209.85.128.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="UVwpk8KY" Received: by mail-wm1-f42.google.com with SMTP id 5b1f17b1804b1-428e1915e18so23587955e9.1 for ; Sat, 17 Aug 2024 23:43:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963434; x=1724568234; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=H3v3ACeRnW6fQEuje9seVAm+KH7JNORlBJR9qvXfha0=; b=UVwpk8KYPP8pWhcl7/wl4AsRmrVEfAGH0vL22K7AwbstGoKhRGYdTruK5ncwZmAHBv rNVQ8cHisauIwAec4eH2/a36K2f+aFkh898mLQc7TxgCE72N+th/zWAPlia5NgYnCMvY oeAmQsWkz7YxiU+UfFy1IU+bXC7EvkM0iu2bL86EUQ4UK6Am25g1Rb6EwT1fsdqAXlr3 56ZEqU1nwgOSOvHzVljrAUzWF9yZQwAX/+2I6cSShPezEl8ZA0epO9G9onLq3KTzqIfx Vu1NpQ5JwXWIBZ9eACPcsnPNkgyimvS4mrW3zDxm2h6Y4OxCIY5hLMJDvSUZcr7E/mPH FFZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963434; x=1724568234; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=H3v3ACeRnW6fQEuje9seVAm+KH7JNORlBJR9qvXfha0=; b=Td9Aq2SxffCpMjAaAJsTWCPwUqc7DQ/t7fI5oe3quifcQ28n3jeKY2H7/H8QY8h+JC X/H/lhQbQ6sdytQvlMUKIvRJyj+EtVR5GU/jvs3Cyp6uwHnp6AKN+TBt2K9NEYaskFU8 QDX17pl8uQZVMhlqOauG/XiOcEVtclC3FdC65CmNpMXpUxXqJFTUxxQ1ef4LAJjyNV/u NoonRNflT8tc4k3bF8u5oYLb6/LFJu2jhGghEnl/VVTtY9R6HAT30NKtISMNzaLJOZgr jFK+S/sKcDvZbHkXIoNgxe9PwHmG8rx1CiGnhT/+p5QNQg7I8wIntBhtTdViuK70XDim AVdA== X-Forwarded-Encrypted: i=1; AJvYcCXTvQ77OXNH41ddg9hmA5bHqDU0VWSwtsyyDaV5vXxMHSNUPrjwab+KdMFka+L2PssEC1amnr4f8//MtKmQv6a9wPcksx/1C3VGao81 X-Gm-Message-State: AOJu0YweWrNN+ykHIYJq7yUDz0h2vArp8RnHEzP3Eu7kD53S9ucAsxPR v6Tn6skJMvhqCGfS0p2fkAPzDC/tIqZia5y3rDekn4iItnW2SpUhJKv1RAZtZ2w= X-Google-Smtp-Source: AGHT+IGdpiZerNesAs6xzBEap+g5NepzOjC8cdpysjNsTdJ9hLtJ74EBvU/xnjt3uajkBUnG6SvmcQ== X-Received: by 2002:a05:600c:3509:b0:426:64a2:5362 with SMTP id 5b1f17b1804b1-429ed7a6100mr51627275e9.8.1723963433892; Sat, 17 Aug 2024 23:43:53 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-429ed4a857bsm66904395e9.0.2024.08.17.23.43.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:43:53 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v5 08/13] riscv: Implement xchg8/16() using Zabha Date: Sun, 18 Aug 2024 08:35:33 +0200 Message-Id: <20240818063538.6651-9-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This adds runtime support for Zabha in xchg8/16() operations. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/include/asm/cmpxchg.h | 65 ++++++++++++++++++++------------ 1 file changed, 41 insertions(+), 24 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index 05ba8a8e2ef5..19779d4d134a 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -14,29 +14,41 @@ #include #include =20 -#define __arch_xchg_masked(sc_sfx, prepend, append, r, p, n) \ -({ \ - u32 *__ptr32b =3D (u32 *)((ulong)(p) & ~0x3); \ - ulong __s =3D ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ - ulong __mask =3D GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ - << __s; \ - ulong __newx =3D (ulong)(n) << __s; \ - ulong __retx; \ - ulong __rc; \ - \ - __asm__ __volatile__ ( \ - prepend \ - "0: lr.w %0, %2\n" \ - " and %1, %0, %z4\n" \ - " or %1, %1, %z3\n" \ - " sc.w" sc_sfx " %1, %1, %2\n" \ - " bnez %1, 0b\n" \ - append \ - : "=3D&r" (__retx), "=3D&r" (__rc), "+A" (*(__ptr32b)) \ - : "rJ" (__newx), "rJ" (~__mask) \ - : "memory"); \ - \ - r =3D (__typeof__(*(p)))((__retx & __mask) >> __s); \ +#define __arch_xchg_masked(sc_sfx, swap_sfx, prepend, sc_append, \ + swap_append, r, p, n) \ +({ \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZABHA)) { \ + __asm__ __volatile__ ( \ + prepend \ + " amoswap" swap_sfx " %0, %z2, %1\n" \ + swap_append \ + : "=3D&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + } else { \ + u32 *__ptr32b =3D (u32 *)((ulong)(p) & ~0x3); \ + ulong __s =3D ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ + ulong __mask =3D GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ + << __s; \ + ulong __newx =3D (ulong)(n) << __s; \ + ulong __retx; \ + ulong __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr.w %0, %2\n" \ + " and %1, %0, %z4\n" \ + " or %1, %1, %z3\n" \ + " sc.w" sc_sfx " %1, %1, %2\n" \ + " bnez %1, 0b\n" \ + sc_append \ + : "=3D&r" (__retx), "=3D&r" (__rc), "+A" (*(__ptr32b)) \ + : "rJ" (__newx), "rJ" (~__mask) \ + : "memory"); \ + \ + r =3D (__typeof__(*(p)))((__retx & __mask) >> __s); \ + } \ }) =20 #define __arch_xchg(sfx, prepend, append, r, p, n) \ @@ -59,8 +71,13 @@ \ switch (sizeof(*__ptr)) { \ case 1: \ + __arch_xchg_masked(sc_sfx, ".b" swap_sfx, \ + prepend, sc_append, swap_append, \ + __ret, __ptr, __new); \ + break; \ case 2: \ - __arch_xchg_masked(sc_sfx, prepend, sc_append, \ + __arch_xchg_masked(sc_sfx, ".h" swap_sfx, \ + prepend, sc_append, swap_append, \ __ret, __ptr, __new); \ break; \ case 4: \ --=20 2.39.2 From nobody Mon Feb 9 04:31:46 2026 Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7BF3CF9FE for ; Sun, 18 Aug 2024 06:44:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963499; cv=none; b=QtxXBO3/7+gNJMyLNlWEkTvEaHMRSNjPfIXPDfrPAFU3xxjkcX5jseMYnalELU9aeUrfZ2mmlQFzycwD3A3Tp/bsZPk520amJA3/6Ui/FDh0+dtfIG49qU6Ed5fmGuADmV1mK3pLO7r0q27x+zTDcvnAjkwJeQ7daQ89JKv4zF0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963499; c=relaxed/simple; bh=XhPMIf8E5h7ieBT3eV+7JhnNWaDTpV0oRSqL+1+Yoms=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=VsYPkzo2tyLEGs6aq8vHVRwQUkbvn3ZK0cMtkObu4RaK2wTqteLue/ddjKEspdufIr40Ftzsf7iErkXE8AiX32FzlbjaNK1DBrlYgmRfmxoOy+lz3e4dUM9Ylc54bMw8ywHp8kigvk/k/jRZquJkut6gqOzU91s2MUTz59YoPRc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=bQk1Abem; arc=none smtp.client-ip=209.85.128.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="bQk1Abem" Received: by mail-wm1-f53.google.com with SMTP id 5b1f17b1804b1-429da8b5feaso34789225e9.2 for ; Sat, 17 Aug 2024 23:44:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963496; x=1724568296; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RFL/O3acOIAnvUQyUxNex19UAi4aS/YbFByq84YgWQ0=; b=bQk1AbempOzqW4v1ovhjBszbUagtUn9Zv96ifPexKqXUxjWpyVO65hpcNUc1ENLJru q7Yb3w5pygHNaQagttJyigqfCm132Ys200UOKP5jwaI1skM8rRxUHfq2zyYT+6rlWVDd UWJaoJhVtp2XKmV7wYJazVMLqBxIEwvkWHpA7Qb8C3YyPkknCteJlT2Dk4yG98ZBoEdz yRgW/POf0m8p3v/iyzPNv19ZQnhCadY+u4L+bL3fU1HJxdOvurygRrE9oJXbtRjO6wlB p12bBpTyEqvhLzaMcTzLwAmnGwv9QT9HM4XNe1kxL7vkCnPqbVlAYHSTB8+2h0/EvqyY Cy5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963496; x=1724568296; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RFL/O3acOIAnvUQyUxNex19UAi4aS/YbFByq84YgWQ0=; b=f4rNVdyHL92+qq/CmQskvIOPayZ+ne7ewH0Oqb3sSYxSd66HgOTkw1rI5hCqpPPkrd vuzmo1uevv0HLt+IA2ocnv2WwhL9CQEQIfcYfQh7JxkohLt2xyHM/F/SQdTbs7YHOSpA Ny06FfrzuCekRUqaZ9QcRJhCCgIN3GLAaQfDeEQmrJLN/r/CuMWqOd1MF4Oe2GiHF3XU xUV6w+542amc1SKnGvrJx399M+N44fpG6m7/QulJEnPxenQNpZCU+4aIKeqRoyPvO3yP oSZnXkhlZUtXOexhpn19BfGQznxCE26eOxABkPNkygnb/56GSx2ZCvYNySmnQU7AvE20 lP1g== X-Forwarded-Encrypted: i=1; AJvYcCVuPBJDGVRrfzzVUGU5ZFqeAtXY35ylTi1K2X2m8rusiy+tFjYOe8BwyDG8BwgH2Kz4uh0RiZzu3K2AWF3lPY5YZms3IsdsL5AAIlgf X-Gm-Message-State: AOJu0YwsS1QVq1yG0ODg1NWzwZRfS5AmRdmbqLoVQs+kswkkaCg1Q37n SlBhCxUlt4uMkhhFlkA3DPztU9T3H+ko2AhRIONI3nRHpn8Qd3+kN5D7fz0OaBE= X-Google-Smtp-Source: AGHT+IFPBW1mn5KeYnZcFZpLT4fLpEaOfFIUvQMa/GHHDd1nNhnkQYgHstBKdHhKt1mlaNsMcxCEDw== X-Received: by 2002:a05:600c:4ed2:b0:426:5b44:2be7 with SMTP id 5b1f17b1804b1-429ed79568amr68241915e9.10.1723963495566; Sat, 17 Aug 2024 23:44:55 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-429ded180dcsm120623885e9.4.2024.08.17.23.44.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:44:55 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Guo Ren Subject: [PATCH v5 09/13] asm-generic: ticket-lock: Reuse arch_spinlock_t of qspinlock Date: Sun, 18 Aug 2024 08:35:34 +0200 Message-Id: <20240818063538.6651-10-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Guo Ren The arch_spinlock_t of qspinlock has contained the atomic_t val, which satisfies the ticket-lock requirement. Thus, unify the arch_spinlock_t into qspinlock_types.h. This is the preparation for the next combo spinlock. Reviewed-by: Leonardo Bras Suggested-by: Arnd Bergmann Link: https://lore.kernel.org/linux-riscv/CAK8P3a2rnz9mQqhN6-e0CGUUv9rntREL= Fdxt_weiD7FxH7fkfQ@mail.gmail.com/ Signed-off-by: Guo Ren Signed-off-by: Guo Ren Acked-by: Peter Zijlstra (Intel) Reviewed-by: Andrew Jones --- include/asm-generic/spinlock.h | 14 +++++++------- include/asm-generic/spinlock_types.h | 12 ++---------- 2 files changed, 9 insertions(+), 17 deletions(-) diff --git a/include/asm-generic/spinlock.h b/include/asm-generic/spinlock.h index 90803a826ba0..4773334ee638 100644 --- a/include/asm-generic/spinlock.h +++ b/include/asm-generic/spinlock.h @@ -32,7 +32,7 @@ =20 static __always_inline void arch_spin_lock(arch_spinlock_t *lock) { - u32 val =3D atomic_fetch_add(1<<16, lock); + u32 val =3D atomic_fetch_add(1<<16, &lock->val); u16 ticket =3D val >> 16; =20 if (ticket =3D=3D (u16)val) @@ -46,31 +46,31 @@ static __always_inline void arch_spin_lock(arch_spinloc= k_t *lock) * have no outstanding writes due to the atomic_fetch_add() the extra * orderings are free. */ - atomic_cond_read_acquire(lock, ticket =3D=3D (u16)VAL); + atomic_cond_read_acquire(&lock->val, ticket =3D=3D (u16)VAL); smp_mb(); } =20 static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) { - u32 old =3D atomic_read(lock); + u32 old =3D atomic_read(&lock->val); =20 if ((old >> 16) !=3D (old & 0xffff)) return false; =20 - return atomic_try_cmpxchg(lock, &old, old + (1<<16)); /* SC, for RCsc */ + return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RC= sc */ } =20 static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) { u16 *ptr =3D (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); - u32 val =3D atomic_read(lock); + u32 val =3D atomic_read(&lock->val); =20 smp_store_release(ptr, (u16)val + 1); } =20 static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) { - u32 val =3D lock.counter; + u32 val =3D lock.val.counter; =20 return ((val >> 16) =3D=3D (val & 0xffff)); } @@ -84,7 +84,7 @@ static __always_inline int arch_spin_is_locked(arch_spinl= ock_t *lock) =20 static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) { - u32 val =3D atomic_read(lock); + u32 val =3D atomic_read(&lock->val); =20 return (s16)((val >> 16) - (val & 0xffff)) > 1; } diff --git a/include/asm-generic/spinlock_types.h b/include/asm-generic/spi= nlock_types.h index 8962bb730945..f534aa5de394 100644 --- a/include/asm-generic/spinlock_types.h +++ b/include/asm-generic/spinlock_types.h @@ -3,15 +3,7 @@ #ifndef __ASM_GENERIC_SPINLOCK_TYPES_H #define __ASM_GENERIC_SPINLOCK_TYPES_H =20 -#include -typedef atomic_t arch_spinlock_t; - -/* - * qrwlock_types depends on arch_spinlock_t, so we must typedef that befor= e the - * include. - */ -#include - -#define __ARCH_SPIN_LOCK_UNLOCKED ATOMIC_INIT(0) +#include +#include =20 #endif /* __ASM_GENERIC_SPINLOCK_TYPES_H */ --=20 2.39.2 From nobody Mon Feb 9 04:31:46 2026 Received: from mail-wr1-f50.google.com (mail-wr1-f50.google.com [209.85.221.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EF934101DE for ; Sun, 18 Aug 2024 06:45:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963561; cv=none; b=IC+IQ0+iAsVmHHIMHLw5w0WJZ2DeC2a5PoKYWtLhhC85qyrL+XGBgBZhULFNW1rRG7AhqGQWtriudXtL1XwNnM5qp4PSSid0fKIfLQPKqSakbFttBeTja4cVGXhqWcLD7IFdQ6NHP8ddhVWGcBOPBz/cV/2JLiSTVRNzsEnkF5g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963561; c=relaxed/simple; bh=qPpipNWa0bJ3ZFtgez7ow/f7HIXA18TVHj9XSMnwy/8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=nX9t3vxYQRD38sQwFaWuK82tL1MS502t3/9qkelzNi5QQruWxY0oBhChPBLNtG1DU4TsQltwrj4nF8dbuz8ERzSrQjmzFPWavLfPdRi7mGmMEgU9B9PxYJGJk1NK0yn7sblJMEGn2NAP7s9yZfTRxAVDBAdnwpW7C7p09SOGDQ4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=Sj4ywXs4; arc=none smtp.client-ip=209.85.221.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="Sj4ywXs4" Received: by mail-wr1-f50.google.com with SMTP id ffacd0b85a97d-3719896b7c8so1004434f8f.3 for ; Sat, 17 Aug 2024 23:45:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963557; x=1724568357; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=J1DJp3g8ecAC/DPl4n2lm0ysWSqC430kG0AnG9TGYTU=; b=Sj4ywXs41OGQ5t/POV7UqzDOAwkk3D+KK037qQfnzZau1eTeUKp8Hlptp0PsunxrxX L22PfFU30vI365Dht9qtJuTkYiuhmvVL8bnFUtEkCilQjkyzNBoT4i83DnYlAPvhY0Vk OcjGY606ZCOHxOnzSMivgnGD+LGZTruh9SqIdwDPB0IDSKIv6vp7HGSbozlVYixWP4FO ygf8BeBXISmFs357kCgUL4IEz/GuerPUs0drKNqieJ1C1ZfrY/P7AjN0jZq4WwkGo+Cx GN741ZCyTjv1IxlRsCdh4P+17bycySfS9oYIyyOUAum0qw4vjW/+uLI/iLiTM5/djgvd iX5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963557; x=1724568357; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=J1DJp3g8ecAC/DPl4n2lm0ysWSqC430kG0AnG9TGYTU=; b=Z2KU4m0ONoDxYG0vd/3bn6m+HSm8NMj0jX8w+rj0fPQ0kTZCKlL0tbt6fcx8xyychw 97gDzuPze6uGPQ3Np9McQnG3ogDQGk300I2KqjJTaQxklSgBj2EEybWz0sUVrJpJcjF6 v8UdN4lvCQJor29iTyQRie4E9ELty+7/3Vz+4G4AaqdIwMCc0TwSnER74LjGTraNrLd0 NNabaX5xoJtbaMYMyr99Shkatn5WTDmDIxKI8biAsV4WktBZGK6STNzNWMh0OsaO04d3 X/3puRu6sKih8fcX9INq1s3PrJbWNkO0PfizibnPqm4w85rsHG0WrIGh2c26VeZEgSZu 16VQ== X-Forwarded-Encrypted: i=1; AJvYcCUmyMYop9kdToiUNyYXUhZcxhS1MmRjq/BnNQBpuremr0ohRIBX2evNPGiWmS/F7knTJF3ELkKBE7/6kaWZQNQ0auB55BGmpCQKacPl X-Gm-Message-State: AOJu0YyRoop5lUqYhUcNduJTA+MXQxCWRN7M3oEQ4L6aO/FG802TrvSj 05rzgXjQ+iJucBGZHLK92bhtIFTiam632TLVBAgniNy9FBMsYyhd5CJ93HmI/Ws= X-Google-Smtp-Source: AGHT+IHPeK5VmE03PU4h9/i27orpAxZrVCijc8uGk6SFEjR1/wG06pKyXQz/+RBNMzRG1dmIOL6ROQ== X-Received: by 2002:a5d:6911:0:b0:368:3717:10aa with SMTP id ffacd0b85a97d-371a73dbb2fmr2311786f8f.11.1723963557033; Sat, 17 Aug 2024 23:45:57 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-37189849cdbsm7338955f8f.28.2024.08.17.23.45.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:45:56 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Guo Ren Subject: [PATCH v5 10/13] asm-generic: ticket-lock: Add separate ticket-lock.h Date: Sun, 18 Aug 2024 08:35:35 +0200 Message-Id: <20240818063538.6651-11-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Guo Ren Add a separate ticket-lock.h to include multiple spinlock versions and select one at compile time or runtime. Reviewed-by: Leonardo Bras Suggested-by: Arnd Bergmann Link: https://lore.kernel.org/linux-riscv/CAK8P3a2rnz9mQqhN6-e0CGUUv9rntREL= Fdxt_weiD7FxH7fkfQ@mail.gmail.com/ Signed-off-by: Guo Ren Signed-off-by: Guo Ren Acked-by: Peter Zijlstra (Intel) Reviewed-by: Andrew Jones --- include/asm-generic/spinlock.h | 87 +--------------------- include/asm-generic/ticket_spinlock.h | 103 ++++++++++++++++++++++++++ 2 files changed, 104 insertions(+), 86 deletions(-) create mode 100644 include/asm-generic/ticket_spinlock.h diff --git a/include/asm-generic/spinlock.h b/include/asm-generic/spinlock.h index 4773334ee638..970590baf61b 100644 --- a/include/asm-generic/spinlock.h +++ b/include/asm-generic/spinlock.h @@ -1,94 +1,9 @@ /* SPDX-License-Identifier: GPL-2.0 */ =20 -/* - * 'Generic' ticket-lock implementation. - * - * It relies on atomic_fetch_add() having well defined forward progress - * guarantees under contention. If your architecture cannot provide this, = stick - * to a test-and-set lock. - * - * It also relies on atomic_fetch_add() being safe vs smp_store_release() = on a - * sub-word of the value. This is generally true for anything LL/SC althou= gh - * you'd be hard pressed to find anything useful in architecture specifica= tions - * about this. If your architecture cannot do this you might be better off= with - * a test-and-set. - * - * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and = hence - * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along= with - * a full fence after the spin to upgrade the otherwise-RCpc - * atomic_cond_read_acquire(). - * - * The implementation uses smp_cond_load_acquire() to spin, so if the - * architecture has WFE like instructions to sleep instead of poll for word - * modifications be sure to implement that (see ARM64 for example). - * - */ - #ifndef __ASM_GENERIC_SPINLOCK_H #define __ASM_GENERIC_SPINLOCK_H =20 -#include -#include - -static __always_inline void arch_spin_lock(arch_spinlock_t *lock) -{ - u32 val =3D atomic_fetch_add(1<<16, &lock->val); - u16 ticket =3D val >> 16; - - if (ticket =3D=3D (u16)val) - return; - - /* - * atomic_cond_read_acquire() is RCpc, but rather than defining a - * custom cond_read_rcsc() here we just emit a full fence. We only - * need the prior reads before subsequent writes ordering from - * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we - * have no outstanding writes due to the atomic_fetch_add() the extra - * orderings are free. - */ - atomic_cond_read_acquire(&lock->val, ticket =3D=3D (u16)VAL); - smp_mb(); -} - -static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) -{ - u32 old =3D atomic_read(&lock->val); - - if ((old >> 16) !=3D (old & 0xffff)) - return false; - - return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RC= sc */ -} - -static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) -{ - u16 *ptr =3D (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); - u32 val =3D atomic_read(&lock->val); - - smp_store_release(ptr, (u16)val + 1); -} - -static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) -{ - u32 val =3D lock.val.counter; - - return ((val >> 16) =3D=3D (val & 0xffff)); -} - -static __always_inline int arch_spin_is_locked(arch_spinlock_t *lock) -{ - arch_spinlock_t val =3D READ_ONCE(*lock); - - return !arch_spin_value_unlocked(val); -} - -static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) -{ - u32 val =3D atomic_read(&lock->val); - - return (s16)((val >> 16) - (val & 0xffff)) > 1; -} - +#include #include =20 #endif /* __ASM_GENERIC_SPINLOCK_H */ diff --git a/include/asm-generic/ticket_spinlock.h b/include/asm-generic/ti= cket_spinlock.h new file mode 100644 index 000000000000..cfcff22b37b3 --- /dev/null +++ b/include/asm-generic/ticket_spinlock.h @@ -0,0 +1,103 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * 'Generic' ticket-lock implementation. + * + * It relies on atomic_fetch_add() having well defined forward progress + * guarantees under contention. If your architecture cannot provide this, = stick + * to a test-and-set lock. + * + * It also relies on atomic_fetch_add() being safe vs smp_store_release() = on a + * sub-word of the value. This is generally true for anything LL/SC althou= gh + * you'd be hard pressed to find anything useful in architecture specifica= tions + * about this. If your architecture cannot do this you might be better off= with + * a test-and-set. + * + * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and = hence + * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along= with + * a full fence after the spin to upgrade the otherwise-RCpc + * atomic_cond_read_acquire(). + * + * The implementation uses smp_cond_load_acquire() to spin, so if the + * architecture has WFE like instructions to sleep instead of poll for word + * modifications be sure to implement that (see ARM64 for example). + * + */ + +#ifndef __ASM_GENERIC_TICKET_SPINLOCK_H +#define __ASM_GENERIC_TICKET_SPINLOCK_H + +#include +#include + +static __always_inline void ticket_spin_lock(arch_spinlock_t *lock) +{ + u32 val =3D atomic_fetch_add(1<<16, &lock->val); + u16 ticket =3D val >> 16; + + if (ticket =3D=3D (u16)val) + return; + + /* + * atomic_cond_read_acquire() is RCpc, but rather than defining a + * custom cond_read_rcsc() here we just emit a full fence. We only + * need the prior reads before subsequent writes ordering from + * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we + * have no outstanding writes due to the atomic_fetch_add() the extra + * orderings are free. + */ + atomic_cond_read_acquire(&lock->val, ticket =3D=3D (u16)VAL); + smp_mb(); +} + +static __always_inline bool ticket_spin_trylock(arch_spinlock_t *lock) +{ + u32 old =3D atomic_read(&lock->val); + + if ((old >> 16) !=3D (old & 0xffff)) + return false; + + return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RC= sc */ +} + +static __always_inline void ticket_spin_unlock(arch_spinlock_t *lock) +{ + u16 *ptr =3D (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); + u32 val =3D atomic_read(&lock->val); + + smp_store_release(ptr, (u16)val + 1); +} + +static __always_inline int ticket_spin_value_unlocked(arch_spinlock_t lock) +{ + u32 val =3D lock.val.counter; + + return ((val >> 16) =3D=3D (val & 0xffff)); +} + +static __always_inline int ticket_spin_is_locked(arch_spinlock_t *lock) +{ + arch_spinlock_t val =3D READ_ONCE(*lock); + + return !ticket_spin_value_unlocked(val); +} + +static __always_inline int ticket_spin_is_contended(arch_spinlock_t *lock) +{ + u32 val =3D atomic_read(&lock->val); + + return (s16)((val >> 16) - (val & 0xffff)) > 1; +} + +/* + * Remapping spinlock architecture specific functions to the corresponding + * ticket spinlock functions. + */ +#define arch_spin_is_locked(l) ticket_spin_is_locked(l) +#define arch_spin_is_contended(l) ticket_spin_is_contended(l) +#define arch_spin_value_unlocked(l) ticket_spin_value_unlocked(l) +#define arch_spin_lock(l) ticket_spin_lock(l) +#define arch_spin_trylock(l) ticket_spin_trylock(l) +#define arch_spin_unlock(l) ticket_spin_unlock(l) + +#endif /* __ASM_GENERIC_TICKET_SPINLOCK_H */ --=20 2.39.2 From nobody Mon Feb 9 04:31:46 2026 Received: from mail-wr1-f49.google.com (mail-wr1-f49.google.com [209.85.221.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9101CFC02 for ; Sun, 18 Aug 2024 06:47:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963622; cv=none; b=Lr/l6yGRtaS7LKk5g/IF0VBO0SlTOKiF9UqvOL7bQaa7DHpQwXCio9/sWmSnepSX06bD0ILXLnqvfbk+ogmyKYbZmGo7n+akkymc/6CpgQf12b2AxkmHCQoJ14SuYs5mUlSaP1gD7KGfjL51ZYugm0V/LtiJRPM2wVOlE5RQkOs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963622; c=relaxed/simple; bh=65N1YDYmlpEvDTK31Deo5Fyg9F/mpaT1yLpXU+1oUBM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Mpsf0h1rxQYznK4FSDSu34r19KN+Ls/LoYIFJU6lfwnj8PgBxwcXJdsyithfAfbMcrKQowMz14gnQVKEQvJi4OjN24eeLzeMWaohok94y064Gj5OAzyStr29FviECndW09Fm4t4hF9/8PDUKuZsGXsIaYWERZ/sUDKVD8DBx2qg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=nes0/oKc; arc=none smtp.client-ip=209.85.221.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="nes0/oKc" Received: by mail-wr1-f49.google.com with SMTP id ffacd0b85a97d-371b098e699so686915f8f.2 for ; Sat, 17 Aug 2024 23:47:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963619; x=1724568419; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vlqxfa1Dgyr/RCZKCOdmB4jO+WlSQVOulpijR1tIjck=; b=nes0/oKcOBBBsfpaVKOoE3KX0Ydi2gRP19gxbtxc03GUOF5MUdL94i+J2+A26H0fHK t7ESmr9UpHJrrJO2REL9xtdCKGfLd8BW2VzBolDQuf1uoR+/DbqqlccDtagvLtTzqz20 Me/MIk9hE0pfGj8d61jPnQqJOMPrbkwq10g4dwBx2fwnsxSU3lpX0HWfbfwbnhmIe1ns kNv8+WbLwI4K2TIJK2RQBZbIjuJFfklA8ILGCaSuy0gxjZKhTOWooHbT28Ged9jhB30W 8tlqWV/lrmI+ni/+gTFqhgRL0cFNs2bC0T0wKJD0HiS6n34jr7151Taw47yyiwp0M8Oy 3VFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963619; x=1724568419; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vlqxfa1Dgyr/RCZKCOdmB4jO+WlSQVOulpijR1tIjck=; b=g7QR9C5embFBHx84cZeqGooMI8LJVUg0Ig62x0HoTnVa3+Q+vvZAUQ2LSj+ETLAF99 Caikeq3lr+ixMZyGb/hQuEVxd9stUuEP4cSAIy2qdA6ddmmQDdTy8AXXe3Fy5p9yXfQF VY/urBOUJwRQN/k36kGoOqacTm4BRR7iJSvvbJCyQNlBJC0gPKAfDXiaW3JK6j4dLywA vSmSY1hmGYPFG6f9PhqSusznJin6PuA8JEVDdvc5GvCLxoqI+aTM9zvdJiKO8hsNOUEd Pa0Pif1GUXuzjNCv6bc4IVt4TGo+RZoyXAzhZHTfoE5BtAEZEmXkNW1tuzcM7/I9Lcos Es4Q== X-Forwarded-Encrypted: i=1; AJvYcCUhRjF6CTL2RQZImzBCsAnUbj/FH/5DSfVdQu5XArmOz5sYTbDHb5iiKb/FTFoIi1Trthd5djuh49lVtstFgPtM4LJ0sYL9zJtNTq3O X-Gm-Message-State: AOJu0Yx7kjdLgk4nA5mXfKz8URReoDx3hMLvjEMrQj6CcF4YLAB+ZjMb Oi0b8Ee006JsBMfcadUqv9QVbNESmd/Oxeg+HzWHu+YlK2Z8vqi9zoR49ozFxmI= X-Google-Smtp-Source: AGHT+IFwS3MzzU8TURVJ4Ap0a8/3o7r3oh1yyMOzNik5dKrEar+UZ6Tut2SPaHafSLHiL9m2WkM9gg== X-Received: by 2002:a5d:694a:0:b0:367:dc45:55ab with SMTP id ffacd0b85a97d-371944561f9mr6249467f8f.25.1723963618917; Sat, 17 Aug 2024 23:46:58 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-371898b89bdsm7269358f8f.112.2024.08.17.23.46.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:46:58 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v5 11/13] riscv: Add ISA extension parsing for Ziccrse Date: Sun, 18 Aug 2024 08:35:36 +0200 Message-Id: <20240818063538.6651-12-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add support to parse the Ziccrse string in the riscv,isa string. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/kernel/cpufeature.c | 1 + 2 files changed, 2 insertions(+) diff --git a/arch/riscv/include/asm/hwcap.h b/arch/riscv/include/asm/hwcap.h index f5d53251c947..9e228b079a6d 100644 --- a/arch/riscv/include/asm/hwcap.h +++ b/arch/riscv/include/asm/hwcap.h @@ -93,6 +93,7 @@ #define RISCV_ISA_EXT_ZCMOP 84 #define RISCV_ISA_EXT_ZAWRS 85 #define RISCV_ISA_EXT_ZABHA 86 +#define RISCV_ISA_EXT_ZICCRSE 87 =20 #define RISCV_ISA_EXT_XLINUXENVCFG 127 =20 diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index 67ebcc4c9424..ea9c255bbe3d 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -314,6 +314,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] =3D { riscv_ext_zicbom_validate), __RISCV_ISA_EXT_SUPERSET_VALIDATE(zicboz, RISCV_ISA_EXT_ZICBOZ, riscv_xli= nuxenvcfg_exts, riscv_ext_zicboz_validate), + __RISCV_ISA_EXT_DATA(ziccrse, RISCV_ISA_EXT_ZICCRSE), __RISCV_ISA_EXT_DATA(zicntr, RISCV_ISA_EXT_ZICNTR), __RISCV_ISA_EXT_DATA(zicond, RISCV_ISA_EXT_ZICOND), __RISCV_ISA_EXT_DATA(zicsr, RISCV_ISA_EXT_ZICSR), --=20 2.39.2 From nobody Mon Feb 9 04:31:46 2026 Received: from mail-wr1-f49.google.com (mail-wr1-f49.google.com [209.85.221.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2B08712B8B for ; Sun, 18 Aug 2024 06:48:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963683; cv=none; b=s/KPv57ve3V4L5ih1FC9X/FfTvA1EKHk1Q2t159kSpuQ7QArmxr6yztlRcLW1FfyoeshinikYi4MwwRwJZgzbm7N/ZRE8jGRNjrVFwtSZWVJYSU6cCI0V9FhJ6e8rnP8Z7KZNyNPcOrYRK9Th6AVH6rK5GOw2oV91tB01V8PaIE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963683; c=relaxed/simple; bh=yIsmDu+znujFYSMd/qij8s9ZhZp0YCl3HXaNmhK3U1E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=iJmBKsFxsoNe2kgj7gjC/VoPbVWySBzeXapPswNzx6BKejha8/AK5XypEeCFs7fqJoi1zNYTvZkK/Uq5k7DcFNxTMkHGIMI9nF7y+vEccdXJV6mf+lZiZnrGPoOpbJIBue0EkyjycP65/pVCbcqU4UgkKYuaNb1KaKj8MospIzw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=DUjnMekm; arc=none smtp.client-ip=209.85.221.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="DUjnMekm" Received: by mail-wr1-f49.google.com with SMTP id ffacd0b85a97d-37196229343so1258947f8f.0 for ; Sat, 17 Aug 2024 23:48:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963680; x=1724568480; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AUJVvfa71x+iwF5DF3rrBv6vNd03sOzmOgv8ZqY/vyI=; b=DUjnMekmsptAb+IIPJdJ7L5zWLyRHw5Mz4QlBSQDK5gX2Y4X+QNlBwrl6WBqNpAfTt 6cpSFyRIvkFeXpsA4R55+YKzF47HmO755FmHIz5f6iDulw5r4CtGOn4WRkcVO3/Fflws 84P/9+ypW835ccX6f4sPhQQABQS1gAe0+u143H5MT7BMzvTsfC+QNgSO5KHlU5jPLR9K QWqxhE5syuH+I1MYMlhGpfCAunrtg6Ye6p3T6JsByOIcRczweaz8yYd70oYfbQ9yCfHs oMdaLP9pfTOV+FteiQwlBoZ4XGNFpQWzITwBfaAnPsEyFdNi82JiK+7RtLnzCR78Q2bn fXxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963680; x=1724568480; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AUJVvfa71x+iwF5DF3rrBv6vNd03sOzmOgv8ZqY/vyI=; b=ctPBCZbDbVEC7ZUq0XWxcu4iguC7bV1aVo5qEXSIb1XPHIUH7UhvEhMi+rBnqcCW4t 0KvkaB+c2Y/l2X6QgbhszFcldBLt+BRIBgpKqqe6kvxZ0rnOygbvIK63ZDFrFPyFiZ18 pW+uI+HOXDinHbHpR6K/4LOhh2ozNHrA5CcKSW7Dh1umtLhJTsdspbzgdchOSz74Jpkc Upo13r1CVevQDskncBxIKtfBgh726ISCITGZgsPjMgrW1V+pDaO49uSzrqbg1G18NevM EtXwksoQqrXHLKXBtVCJcdEdN48OZV9bKYf8/urrDFaxgxF09prKBHiwjxwyDcDM5kcG 9OPQ== X-Forwarded-Encrypted: i=1; AJvYcCUVvhBpJzNuDBFxu0430pP/I/9k7U+Yfss2r7UY7QpB4ndx5ia2zUpd3hNuwpaP1pLxt7PboIFuQLh421Ge7hoAyhJX/2QmaMLPm7Fr X-Gm-Message-State: AOJu0YzLNdNDIm5dFdmpa/ebejb3DJJS1Fp9i6LR9Q7rs5dmE9GYSFUX +H6NTfs6wP9e6yrvLZnMrycXM/yGwSYnhpw5aOc71/Enm8sjehU+0fZrhXO+aJU= X-Google-Smtp-Source: AGHT+IHGhKQ874lL1J7nsjPh6FSfKHuvYCqLot3YTtCsdD7yaCkb97X7rPbK0sQldMNV0hrVqhCG+Q== X-Received: by 2002:a5d:53c5:0:b0:367:8a00:fac3 with SMTP id ffacd0b85a97d-371946514e9mr5069861f8f.30.1723963680234; Sat, 17 Aug 2024 23:48:00 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3718983a1b8sm7336054f8f.5.2024.08.17.23.47.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:47:59 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v5 12/13] dt-bindings: riscv: Add Ziccrse ISA extension description Date: Sun, 18 Aug 2024 08:35:37 +0200 Message-Id: <20240818063538.6651-13-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add description for the Ziccrse ISA extension which was ratified in the riscv profiles specification v1.0. Signed-off-by: Alexandre Ghiti Reviewed-by: Guo Ren Acked-by: Conor Dooley Reviewed-by: Andrew Jones --- Documentation/devicetree/bindings/riscv/extensions.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Documentation/devicetree/bindings/riscv/extensions.yaml b/Docu= mentation/devicetree/bindings/riscv/extensions.yaml index a63578b95c4a..4f174c4c08ff 100644 --- a/Documentation/devicetree/bindings/riscv/extensions.yaml +++ b/Documentation/devicetree/bindings/riscv/extensions.yaml @@ -289,6 +289,12 @@ properties: in commit 64074bc ("Update version numbers for Zfh/Zfinx") of riscv-isa-manual. =20 + - const: ziccrse + description: + The standard Ziccrse extension which provides forward progress + guarantee on LR/SC sequences, as ratified in commit b1d806605f= 87 + ("Updated to ratified state.") of the riscv profiles specifica= tion. + - const: zk description: The standard Zk Standard Scalar cryptography extension as rati= fied --=20 2.39.2 From nobody Mon Feb 9 04:31:46 2026 Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7CA8617597 for ; Sun, 18 Aug 2024 06:49:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963745; cv=none; b=OiFspz4smehxr7a+4E8hpA+tiXRVv6enNsJ5INOaRT5v5Psz2veVvsfWGC6Z/P7hYPJAgeod6U3EsDdWNJQ1yUatmUL9qaNxy8Z6PMNMBxFQAHM0HKW/wZGRItC10fDwD9ud5mOKozN8hh3zjbjZO3tbgbTx/pFNQWawyEOKB0A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723963745; c=relaxed/simple; bh=evPt/Nz5zuNchescsRSsW9NnGKR8QvlKr7R+ghjIN/8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=I8EJqMihui0qnmQJJnh6FmJ11lkY92pkemliUjpj7vau6GMtfqfuWEGuBU65BvkHhuQF43u5OgyUR40EyQKm2oE4gcymRx7zEa1CG/4UuiVDKXEwr6OUmmj15mvK4jMFxwRr4B8mjV13lCYxJlToY4pSeMcE/vSYSZ1uCqXJzHM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=h1MWBpGv; arc=none smtp.client-ip=209.85.128.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="h1MWBpGv" Received: by mail-wm1-f48.google.com with SMTP id 5b1f17b1804b1-429d2d7be1eso16609385e9.1 for ; Sat, 17 Aug 2024 23:49:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963742; x=1724568542; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=luYpQp4OVqMmof3eSGbnJOpDEcugh3MQdyInZ+zIklA=; b=h1MWBpGvwgi0STP0YtnPVWTRtTFgQ1OKuFixWKg2pw0Yv3T6cdHx7bmEYXdJne3Lgr S3bwOVTXo+VnSQLZpMWDX3gSvH0LIFUu0oxd3hdaQb9CO4NpuXVvZWKSKCof/pM+exSL m7ZjR6ut4TjvLWNYWlpec5MzircqExMHAf+baUiPFHZ5kL5XqjAwd1gbAyHtvZragQxL khgZ07nZMXWuwJ+zK5N5/pQR+dMmxty5epYN5SQFjtp1FcHwEy9tQjCYySC5zaENWkkw SS3prX55coCF3N3bmziRpEsZscgC5dwWUBxFmySyqCU00dJ4ZCblYbQPxUy4A3gr4dVE fATw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963742; x=1724568542; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=luYpQp4OVqMmof3eSGbnJOpDEcugh3MQdyInZ+zIklA=; b=gk34nzaweLdu4/B01HNW5j8QA/pURwIcu6JexJdVqy2YdGvnYauDx88lhNd3wNjcry wjO0WXXBHRaroQJVGHZd05p2JCQsqXMHFHv/b73Z1FnPr8XZVhaaJTeloYWUDAoqpnUu l12uS9svj/f+WwQmdzkJqitMQp9YPGtKOk80fvyeTdvKqUQScOaoT85xSAHLXvEr1QXU WXamDI/ESUnJlKmG0S72uohxRgMyy+8FxQc5Xf72IDZi34JTzR2XQ+794MyLfWKvNUqJ Z8Ti8EXQKjCx0pYx0PS3A4svE3rhye3KTLve3w7hERpwPw/Fq61Ji4tBnvQxoppSiFXf SwYg== X-Forwarded-Encrypted: i=1; AJvYcCWB7hPxVudJieXLKWHWxj1WVB0VKvSFM63wZA2gRxheYgA+2po1Xmtij6R+3sW+K2gQigX+PvN2ws7SXMd0mBz5Q3Zv8NUSLDGFx9GW X-Gm-Message-State: AOJu0Yzcmu/DyF5b6gCm3PB+cIvD7iJOSuKeu0Tqo5k8k9IW7az110FN iySYxnpFxBN2o0LDGWTiVDTdumj/xOhXq/e3HRatzURsL7nPeaImHUjxSw8hjQQ= X-Google-Smtp-Source: AGHT+IEsvkhvyT/0ituMtpPmeaM6FNA/kWbI3AHcBR+0rlpsw+92Yz8DA6J8n1o3o1wi0guN4MhOfg== X-Received: by 2002:a05:600c:4e91:b0:426:64c1:8388 with SMTP id 5b1f17b1804b1-429eda23e74mr44379405e9.17.1723963741667; Sat, 17 Aug 2024 23:49:01 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-429ed648f0csm66770755e9.9.2024.08.17.23.49.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:49:01 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v5 13/13] riscv: Add qspinlock support Date: Sun, 18 Aug 2024 08:35:38 +0200 Message-Id: <20240818063538.6651-14-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In order to produce a generic kernel, a user can select CONFIG_COMBO_SPINLOCKS which will fallback at runtime to the ticket spinlock implementation if Zabha or Ziccrse are not present. Note that we can't use alternatives here because the discovery of extensions is done too late and we need to start with the qspinlock implementation because the ticket spinlock implementation would pollute the spinlock value, so let's use static keys. This is largely based on Guo's work and Leonardo reviews at [1]. Link: https://lore.kernel.org/linux-riscv/20231225125847.2778638-1-guoren@k= ernel.org/ [1] Signed-off-by: Guo Ren Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- .../locking/queued-spinlocks/arch-support.txt | 2 +- arch/riscv/Kconfig | 34 ++++++++++++++ arch/riscv/include/asm/Kbuild | 4 +- arch/riscv/include/asm/spinlock.h | 47 +++++++++++++++++++ arch/riscv/kernel/setup.c | 37 +++++++++++++++ include/asm-generic/qspinlock.h | 2 + include/asm-generic/ticket_spinlock.h | 2 + 7 files changed, 126 insertions(+), 2 deletions(-) create mode 100644 arch/riscv/include/asm/spinlock.h diff --git a/Documentation/features/locking/queued-spinlocks/arch-support.t= xt b/Documentation/features/locking/queued-spinlocks/arch-support.txt index 22f2990392ff..cf26042480e2 100644 --- a/Documentation/features/locking/queued-spinlocks/arch-support.txt +++ b/Documentation/features/locking/queued-spinlocks/arch-support.txt @@ -20,7 +20,7 @@ | openrisc: | ok | | parisc: | TODO | | powerpc: | ok | - | riscv: | TODO | + | riscv: | ok | | s390: | TODO | | sh: | TODO | | sparc: | ok | diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index ef55ab94027e..201d0669db7f 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -79,6 +79,7 @@ config RISCV select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP select ARCH_WANTS_NO_INSTR select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE + select ARCH_WEAK_RELEASE_ACQUIRE if ARCH_USE_QUEUED_SPINLOCKS select BINFMT_FLAT_NO_DATA_START_OFFSET if !MMU select BUILDTIME_TABLE_SORT if MMU select CLINT_TIMER if RISCV_M_MODE @@ -488,6 +489,39 @@ config NODES_SHIFT Specify the maximum number of NUMA Nodes available on the target system. Increases memory reserved to accommodate various tables. =20 +choice + prompt "RISC-V spinlock type" + default RISCV_COMBO_SPINLOCKS + +config RISCV_TICKET_SPINLOCKS + bool "Using ticket spinlock" + +config RISCV_QUEUED_SPINLOCKS + bool "Using queued spinlock" + depends on SMP && MMU && NONPORTABLE + select ARCH_USE_QUEUED_SPINLOCKS + help + The queued spinlock implementation requires the forward progress + guarantee of cmpxchg()/xchg() atomic operations: CAS with Zabha or + LR/SC with Ziccrse provide such guarantee. + + Select this if and only if Zabha or Ziccrse is available on your + platform, RISCV_QUEUED_SPINLOCKS must not be selected for platforms + without one of those extensions. + + If unsure, select RISCV_COMBO_SPINLOCKS, which will use qspinlocks + when supported and otherwise ticket spinlocks. + +config RISCV_COMBO_SPINLOCKS + bool "Using combo spinlock" + depends on SMP && MMU + select ARCH_USE_QUEUED_SPINLOCKS + help + Embed both queued spinlock and ticket lock so that the spinlock + implementation can be chosen at runtime. + +endchoice + config RISCV_ALTERNATIVE bool depends on !XIP_KERNEL diff --git a/arch/riscv/include/asm/Kbuild b/arch/riscv/include/asm/Kbuild index 5c589770f2a8..1c2618c964f0 100644 --- a/arch/riscv/include/asm/Kbuild +++ b/arch/riscv/include/asm/Kbuild @@ -5,10 +5,12 @@ syscall-y +=3D syscall_table_64.h generic-y +=3D early_ioremap.h generic-y +=3D flat.h generic-y +=3D kvm_para.h +generic-y +=3D mcs_spinlock.h generic-y +=3D parport.h -generic-y +=3D spinlock.h generic-y +=3D spinlock_types.h +generic-y +=3D ticket_spinlock.h generic-y +=3D qrwlock.h generic-y +=3D qrwlock_types.h +generic-y +=3D qspinlock.h generic-y +=3D user.h generic-y +=3D vmlinux.lds.h diff --git a/arch/riscv/include/asm/spinlock.h b/arch/riscv/include/asm/spi= nlock.h new file mode 100644 index 000000000000..e5121b89acea --- /dev/null +++ b/arch/riscv/include/asm/spinlock.h @@ -0,0 +1,47 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __ASM_RISCV_SPINLOCK_H +#define __ASM_RISCV_SPINLOCK_H + +#ifdef CONFIG_RISCV_COMBO_SPINLOCKS +#define _Q_PENDING_LOOPS (1 << 9) + +#define __no_arch_spinlock_redefine +#include +#include +#include + +/* + * TODO: Use an alternative instead of a static key when we are able to pa= rse + * the extensions string earlier in the boot process. + */ +DECLARE_STATIC_KEY_TRUE(qspinlock_key); + +#define SPINLOCK_BASE_DECLARE(op, type, type_lock) \ +static __always_inline type arch_spin_##op(type_lock lock) \ +{ \ + if (static_branch_unlikely(&qspinlock_key)) \ + return queued_spin_##op(lock); \ + return ticket_spin_##op(lock); \ +} + +SPINLOCK_BASE_DECLARE(lock, void, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(unlock, void, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(is_locked, int, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(is_contended, int, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(trylock, bool, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(value_unlocked, int, arch_spinlock_t) + +#elif defined(CONFIG_RISCV_QUEUED_SPINLOCKS) + +#include + +#else + +#include + +#endif + +#include + +#endif /* __ASM_RISCV_SPINLOCK_H */ diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c index a2cde65b69e9..438e4f6ad2ad 100644 --- a/arch/riscv/kernel/setup.c +++ b/arch/riscv/kernel/setup.c @@ -244,6 +244,42 @@ static void __init parse_dtb(void) #endif } =20 +#if defined(CONFIG_RISCV_COMBO_SPINLOCKS) +DEFINE_STATIC_KEY_TRUE(qspinlock_key); +EXPORT_SYMBOL(qspinlock_key); +#endif + +static void __init riscv_spinlock_init(void) +{ + char *using_ext =3D NULL; + + if (IS_ENABLED(CONFIG_RISCV_TICKET_SPINLOCKS)) { + pr_info("Ticket spinlock: enabled\n"); + return; + } + + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && + IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && + riscv_isa_extension_available(NULL, ZABHA) && + riscv_isa_extension_available(NULL, ZACAS)) { + using_ext =3D "using Zabha"; + } else if (riscv_isa_extension_available(NULL, ZICCRSE)) { + using_ext =3D "using Ziccrse"; + } +#if defined(CONFIG_RISCV_COMBO_SPINLOCKS) + else { + static_branch_disable(&qspinlock_key); + pr_info("Ticket spinlock: enabled\n"); + return; + } +#endif + + if (!using_ext) + pr_err("Queued spinlock without Zabha or Ziccrse"); + else + pr_info("Queued spinlock %s: enabled\n", using_ext); +} + extern void __init init_rt_signal_env(void); =20 void __init setup_arch(char **cmdline_p) @@ -297,6 +333,7 @@ void __init setup_arch(char **cmdline_p) riscv_set_dma_cache_alignment(); =20 riscv_user_isa_enable(); + riscv_spinlock_init(); } =20 bool arch_cpu_is_hotpluggable(int cpu) diff --git a/include/asm-generic/qspinlock.h b/include/asm-generic/qspinloc= k.h index 0655aa5b57b2..bf47cca2c375 100644 --- a/include/asm-generic/qspinlock.h +++ b/include/asm-generic/qspinlock.h @@ -136,6 +136,7 @@ static __always_inline bool virt_spin_lock(struct qspin= lock *lock) } #endif =20 +#ifndef __no_arch_spinlock_redefine /* * Remapping spinlock architecture specific functions to the corresponding * queued spinlock functions. @@ -146,5 +147,6 @@ static __always_inline bool virt_spin_lock(struct qspin= lock *lock) #define arch_spin_lock(l) queued_spin_lock(l) #define arch_spin_trylock(l) queued_spin_trylock(l) #define arch_spin_unlock(l) queued_spin_unlock(l) +#endif =20 #endif /* __ASM_GENERIC_QSPINLOCK_H */ diff --git a/include/asm-generic/ticket_spinlock.h b/include/asm-generic/ti= cket_spinlock.h index cfcff22b37b3..325779970d8a 100644 --- a/include/asm-generic/ticket_spinlock.h +++ b/include/asm-generic/ticket_spinlock.h @@ -89,6 +89,7 @@ static __always_inline int ticket_spin_is_contended(arch_= spinlock_t *lock) return (s16)((val >> 16) - (val & 0xffff)) > 1; } =20 +#ifndef __no_arch_spinlock_redefine /* * Remapping spinlock architecture specific functions to the corresponding * ticket spinlock functions. @@ -99,5 +100,6 @@ static __always_inline int ticket_spin_is_contended(arch= _spinlock_t *lock) #define arch_spin_lock(l) ticket_spin_lock(l) #define arch_spin_trylock(l) ticket_spin_trylock(l) #define arch_spin_unlock(l) ticket_spin_unlock(l) +#endif =20 #endif /* __ASM_GENERIC_TICKET_SPINLOCK_H */ --=20 2.39.2