From nobody Sun Nov 24 17:42:18 2024 Received: from mail-wr1-f42.google.com (mail-wr1-f42.google.com [209.85.221.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C3897156875 for ; Sun, 3 Nov 2024 14:52:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730645580; cv=none; b=GdKp6WhINQEN1QdkMCJ5F/NIQLJpv4rzJzkCmbLT/KfeOCKx4ZGpuJYGKZt1wUWVNjSEgK8gLiJIeqRJQNHR3b0ikcMocusRC5q1h3phJMNm49Mh9rt/OH5UzzV8zJ9FIIkXygZNaioyxFtTCf6BZlP+gGlQULzeQ366A3D3QAA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730645580; c=relaxed/simple; bh=3vAJDoEpGiFg4mI/K3dJV6TDhU2SVBU3QlsfeYAyaSY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gxKYTFzIHLFI7kapTFSn/xyCDBVynFL7gw013fzsti5hv/MzPn72WWi/90weTX0v8yGnyhTfFqvwalnlkfOCh2bJsk4ISMPUR+jz7gnQeal54+xuv1wH9cfNIgbKP8ITtmR+5HtkJzuGaHZJaDtAKmKGRfEBAF/o2CvZxxXy3kM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=S/x173O+; arc=none smtp.client-ip=209.85.221.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="S/x173O+" Received: by mail-wr1-f42.google.com with SMTP id ffacd0b85a97d-37d462c91a9so2057420f8f.2 for ; Sun, 03 Nov 2024 06:52:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730645577; x=1731250377; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ll6PO6fq7jekV3yT3oRS1DwKakMdlEGxiDtEOFhGbwU=; b=S/x173O+dkOdvD4YGsaTN1CRNFqOdxT5f3ODgR+/SNyItHJ4qkbmoGMListD6HYQHt 2GgeZLbHxSV42Jpv5QHPX4lWlAq5d5l/EpiCOATfBK+VpAZPuLu8k/rU0NHy2+c0fo0h EHjnRoOZC0BqUzS5+/9G8/PYIfpklYD0WLVYpLpljzUc9y5dv4+IdUtPXH3wDxhqPug5 rMINQfBWLA5duo3M/tyu85390WfpkV1537JBmp7B8aMxeyCPBMs9RtnHAEENX3LD2XvW 8a0SwlKIvGtnkNMOStA+Y8Q/LXWZPD0zm3PF7Fil91fgfDF8bOiKowwQxYBjfrNOSaJs 6rFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730645577; x=1731250377; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ll6PO6fq7jekV3yT3oRS1DwKakMdlEGxiDtEOFhGbwU=; b=u25pUXrPm837/tfSZjIZfcSSevbiidlrwg5+i6fJ7IKJEybCg5USSqLPgnYnmdi/mt GiaxDOeBhvcQWLrZfEAiRVBPZmDUsGeu3iGffm3OoJsSyJ24owSqL+e/TLAgF9+lIjL+ bZtmJM9m+HFJc82uct8hD0ts+7kpQrsLjyVofrdkMgjUGS633CRQTUBv9cHWca+mD2vO QakQNVP8D3i2FG+Z7tQaugLH6PQJgHYRZA5Fyc5mF4vvwUZZNgVcWsgVsBMDdw8Lq84y mmnBid+8GVOipZLmvWhg0wZ/6d3V6T2es+BCNJ84CJb60QLF2H7R/hYH35EDI2nEe44g 3ODw== X-Forwarded-Encrypted: i=1; AJvYcCVPDVhP16hUWAHdXXC30EpAZ5W0CmKjZtzsZyLQeW4Svj1Y+By2TEStoTVmjD6qKzF6kXjQDeVpgE5jjkk=@vger.kernel.org X-Gm-Message-State: AOJu0YxC1ynN/BeVsR/HRPA3LmpiDu7LRj1et7jXVZuqJDr760XgWNr5 wAE5+s9gi6mdMWUgO7gWvOU4JWMohfbFoWZOYfIddEL6A9YCct6vD44iZJF/MKk= X-Google-Smtp-Source: AGHT+IFvsXKhMz1fjru6A+QnFRydk5FLICcqibf9uwIL5DAG/Ic4jbFEh3bsH3jcltqVkAZyHMwFyA== X-Received: by 2002:a05:6000:1b08:b0:378:89be:1825 with SMTP id ffacd0b85a97d-381b710fbb0mr11387559f8f.49.1730645577054; Sun, 03 Nov 2024 06:52:57 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-381c113e642sm10721990f8f.68.2024.11.03.06.52.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 06:52:56 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v6 01/13] riscv: Move cpufeature.h macros into their own header Date: Sun, 3 Nov 2024 15:51:41 +0100 Message-Id: <20241103145153.105097-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" asm/cmpxchg.h will soon need riscv_has_extension_unlikely() macros and then needs to include asm/cpufeature.h which introduces a lot of header circular dependencies. So move the riscv_has_extension_XXX() macros into their own header which prevents such circular dependencies by including a restricted number of headers. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones Reviewed-by: Andrea Parri --- arch/riscv/include/asm/cpufeature-macros.h | 66 ++++++++++++++++++++++ arch/riscv/include/asm/cpufeature.h | 61 ++------------------ 2 files changed, 70 insertions(+), 57 deletions(-) create mode 100644 arch/riscv/include/asm/cpufeature-macros.h diff --git a/arch/riscv/include/asm/cpufeature-macros.h b/arch/riscv/includ= e/asm/cpufeature-macros.h new file mode 100644 index 000000000000..a8103edbf51f --- /dev/null +++ b/arch/riscv/include/asm/cpufeature-macros.h @@ -0,0 +1,66 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright 2022-2024 Rivos, Inc + */ + +#ifndef _ASM_CPUFEATURE_MACROS_H +#define _ASM_CPUFEATURE_MACROS_H + +#include +#include + +#define STANDARD_EXT 0 + +bool __riscv_isa_extension_available(const unsigned long *isa_bitmap, unsi= gned int bit); +#define riscv_isa_extension_available(isa_bitmap, ext) \ + __riscv_isa_extension_available(isa_bitmap, RISCV_ISA_EXT_##ext) + +static __always_inline bool __riscv_has_extension_likely(const unsigned lo= ng vendor, + const unsigned long ext) +{ + asm goto(ALTERNATIVE("j %l[l_no]", "nop", %[vendor], %[ext], 1) + : + : [vendor] "i" (vendor), [ext] "i" (ext) + : + : l_no); + + return true; +l_no: + return false; +} + +static __always_inline bool __riscv_has_extension_unlikely(const unsigned = long vendor, + const unsigned long ext) +{ + asm goto(ALTERNATIVE("nop", "j %l[l_yes]", %[vendor], %[ext], 1) + : + : [vendor] "i" (vendor), [ext] "i" (ext) + : + : l_yes); + + return false; +l_yes: + return true; +} + +static __always_inline bool riscv_has_extension_unlikely(const unsigned lo= ng ext) +{ + compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_= MAX"); + + if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) + return __riscv_has_extension_unlikely(STANDARD_EXT, ext); + + return __riscv_isa_extension_available(NULL, ext); +} + +static __always_inline bool riscv_has_extension_likely(const unsigned long= ext) +{ + compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_= MAX"); + + if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) + return __riscv_has_extension_likely(STANDARD_EXT, ext); + + return __riscv_isa_extension_available(NULL, ext); +} + +#endif /* _ASM_CPUFEATURE_MACROS_H */ diff --git a/arch/riscv/include/asm/cpufeature.h b/arch/riscv/include/asm/c= pufeature.h index 45f9c1171a48..87ed88fc950d 100644 --- a/arch/riscv/include/asm/cpufeature.h +++ b/arch/riscv/include/asm/cpufeature.h @@ -8,9 +8,11 @@ =20 #include #include +#include +#include +#include #include -#include -#include +#include =20 /* * These are probed via a device_initcall(), via either the SBI or directly @@ -103,61 +105,6 @@ extern const size_t riscv_isa_ext_count; extern bool riscv_isa_fallback; =20 unsigned long riscv_isa_extension_base(const unsigned long *isa_bitmap); - -#define STANDARD_EXT 0 - -bool __riscv_isa_extension_available(const unsigned long *isa_bitmap, unsi= gned int bit); -#define riscv_isa_extension_available(isa_bitmap, ext) \ - __riscv_isa_extension_available(isa_bitmap, RISCV_ISA_EXT_##ext) - -static __always_inline bool __riscv_has_extension_likely(const unsigned lo= ng vendor, - const unsigned long ext) -{ - asm goto(ALTERNATIVE("j %l[l_no]", "nop", %[vendor], %[ext], 1) - : - : [vendor] "i" (vendor), [ext] "i" (ext) - : - : l_no); - - return true; -l_no: - return false; -} - -static __always_inline bool __riscv_has_extension_unlikely(const unsigned = long vendor, - const unsigned long ext) -{ - asm goto(ALTERNATIVE("nop", "j %l[l_yes]", %[vendor], %[ext], 1) - : - : [vendor] "i" (vendor), [ext] "i" (ext) - : - : l_yes); - - return false; -l_yes: - return true; -} - -static __always_inline bool riscv_has_extension_unlikely(const unsigned lo= ng ext) -{ - compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_= MAX"); - - if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) - return __riscv_has_extension_unlikely(STANDARD_EXT, ext); - - return __riscv_isa_extension_available(NULL, ext); -} - -static __always_inline bool riscv_has_extension_likely(const unsigned long= ext) -{ - compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_= MAX"); - - if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) - return __riscv_has_extension_likely(STANDARD_EXT, ext); - - return __riscv_isa_extension_available(NULL, ext); -} - static __always_inline bool riscv_cpu_has_extension_likely(int cpu, const = unsigned long ext) { compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_= MAX"); --=20 2.39.2 From nobody Sun Nov 24 17:42:18 2024 Received: from mail-wr1-f41.google.com (mail-wr1-f41.google.com [209.85.221.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 00810487A5 for ; Sun, 3 Nov 2024 14:53:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730645642; cv=none; b=KsE1+VhYQGcxJroBDd0oe9L5+yk+8EwQbufinOP1vPkHQMT9UbRY8C+2hx5yY+CuB7dt4ks2I3AGR91WA2AFZP/XZguY5SfuJsaMrdMjeC5oJbBSmQ8QEqqvJxOGq5ZrAuecb4HD7xjbINQWKNfsc5zddiaj/RbU0RAR7GDWPgo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730645642; c=relaxed/simple; bh=I9YIkC+Jr1vfCxbUp9hWddXX3XizMNgS5ezZYWoSkNs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=oRGT4t4egbpe0d76UizZlBuuBMk3khx17jmPzRsMPaCuk8G8LicILeI/KBqQYlkHqaKkth6bZiQMUivDpp1tQioLGQHvnZYPYR0hUOz0t8L4Zo5NxZ9Sr8fIeA54y3Ala6pufqlIiEzIxwQSQxYI7d1GiY8Y47flMcb038mxG/Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=IwLOAFDm; arc=none smtp.client-ip=209.85.221.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="IwLOAFDm" Received: by mail-wr1-f41.google.com with SMTP id ffacd0b85a97d-37d43a9bc03so2272042f8f.2 for ; Sun, 03 Nov 2024 06:53:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730645638; x=1731250438; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Nql3DD9359qdXdLLpVtxL8oa+5wIDas5DqswNkA3KmI=; b=IwLOAFDmmiiHzdE9o+zlyEuI1eZQCoqkvnE2g1WFiI5dQwK2UL80wONjuRQdrnbZoa iGXKkYHh+fRmiDei6b3ZIzGouZBWYphaxq5QteihSa0kZEcqUTsUHVBK0eZJaMZa0D2S pMRIZ917kFyxtBtJS49NORVCRFbo42TNBDCQbOh4PvZLrJC2YwMy8c2/3PuD+kLfSxlH Q/u1FFWbRWnOz+UkOrYpLjXroFVbmUHNW7D0mOcW69X86dTJZDDAXwR/N0DFkhNQaQtv hRrDvqMEM6yid6ieQg/EKs4L5t5FdNmbHiNQv/v9ehNcvjBMo9p2ZPI2zYI/uIQgO7cD S75g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730645638; x=1731250438; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Nql3DD9359qdXdLLpVtxL8oa+5wIDas5DqswNkA3KmI=; b=TsNPzijqws0+2VSHFt9200ea5wIErPH+UITxLJAWt8ylpusBLXEqvbRhJUF2+jGQDV YNsotQx4aZs3nT9UDXrOjJN+u1RHJrY4mLokmP2DuvoOtF5S92g/p2Gv7EVlaDbBlZQ6 u8ffXkQfGa2j6AkDSO27OSErjas61N+2F3Al1xSNai4RY6nMJfkX8M3dDC62j2ipF/u/ hKNdDFtMIqHE65Xgnad5Mb1FhuEbY02NwjNQ3lfudiwgKQwwHDk2nLZ+/u8FWI+fjmVp 9RIG3royCeqi26pZTemABnRgZDv9NsYvGP6pNZ+kIDn1R1fXKXRrC+tb4MOkjki4o3QR NJyg== X-Forwarded-Encrypted: i=1; AJvYcCWnKnN3Z101Dnig0ROhI6cl63mkd3qA1vX2l/0BRHMaGANAUkpmE1bzAmFo4oHLBxwAoFYUdBXgmYlMebE=@vger.kernel.org X-Gm-Message-State: AOJu0Yxb+dTEPQ+BhYoHxWyhgMLikntbxofUjc7KxnZtKj03dU0Wrp0L xqUU6LrdThN04NiRJr6WUAwPnOmaspmBGCdsFqRS2tsj5GbVbHk3uwj0HlywX2E= X-Google-Smtp-Source: AGHT+IEzqXU+n6yFRFqvDrou02tN/4/eCcsT/sjowi0L/ehc2kcbHr+rCGMDYfv6CZMwRO5vLoFRDw== X-Received: by 2002:a5d:47ac:0:b0:371:9360:c4a8 with SMTP id ffacd0b85a97d-381c7a472ecmr8333590f8f.6.1730645638302; Sun, 03 Nov 2024 06:53:58 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-381c10d7e0fsm10710932f8f.47.2024.11.03.06.53.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 06:53:57 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v6 02/13] riscv: Do not fail to build on byte/halfword operations with Zawrs Date: Sun, 3 Nov 2024 15:51:42 +0100 Message-Id: <20241103145153.105097-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" riscv does not have lr instructions on byte and halfword but the qspinlock implementation actually uses such atomics provided by the Zabha extension, so those sizes are legitimate. Then instead of failing to build, just fallback to the !Zawrs path. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones Reviewed-by: Andrea Parri --- arch/riscv/include/asm/cmpxchg.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index ebbce134917c..ac1d7df898ef 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -245,6 +245,11 @@ static __always_inline void __cmpwait(volatile void *p= tr, : : : : no_zawrs); =20 switch (size) { + case 1: + fallthrough; + case 2: + /* RISC-V doesn't have lr instructions on byte and half-word. */ + goto no_zawrs; case 4: asm volatile( " lr.w %0, %1\n" --=20 2.39.2 From nobody Sun Nov 24 17:42:18 2024 Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 764F217C234 for ; Sun, 3 Nov 2024 14:55:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730645703; cv=none; b=nG76ZLit6zEumVqIHeJf9Cqr6HNVqXe5nlj4a3/bWtcrWILi+goZvCvFNM38EusQcjOxemVDLrF8M3lgUO/47oQJmn5g4KATipZznxSkFnFslvfPsxz7iHopd8OmIP1sYCVURO2aqXvVXnZsnjnESAbSeRxt7Dooqeq4IMLR140= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730645703; c=relaxed/simple; bh=t/Q7smNje9JU7iFN5RPMXqy/JACg1F/f1IIsKrIvTkw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=FGDm1E2krmJWtDVDhDHcYVmaaE4klJYORrhbVq/yemuwZtLLQmjS1lN7YaCqVg/U5H1E6oO7RwvgS6jOaY1BPtXCA4sHfa7fm9gNRPyTAz/igIumZWiUFnI101w7xsOAYlSgD9TiX+zr//WpFvcd/Vc9txWseAH6IpHKMqr6UVY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=ptuzgnkT; arc=none smtp.client-ip=209.85.221.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="ptuzgnkT" Received: by mail-wr1-f44.google.com with SMTP id ffacd0b85a97d-37d6a2aa748so1882536f8f.1 for ; Sun, 03 Nov 2024 06:55:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730645700; x=1731250500; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=j2RpMtAPESbZvfWQgyCWL6emNu2SZ3U7eDtJoRXd/Zk=; b=ptuzgnkTeLtzI7466k6koYfupE2e5Ft7sLDqL9Zd57GMdT6Ku31E8Q5ThO3A7iYsVe kCEAz7b9axVgGoPZdy9NYXv588ae1pcNOBDkcrILQvDdBOJllkYqgKpI6g3s5zsN6REM vdCluoAyqunccqpe3bZlNcCp4SOtMzKrG0xbOCBpnO49AYxjvdXLT+q6DL2CTcLJxF4J moRGqt7J8xpPf1iv89O9x1GpkUMdBOI1eHmoVEHttZYTo4Su30fvrB5f5PCtE5QOEJGl 2a6C6v95rdROFjWW+9O54vcANImCY1Jhcj/uS3DB9xSAmCkBB0cRv8gO0KG/EaaTH84J 1Jzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730645700; x=1731250500; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=j2RpMtAPESbZvfWQgyCWL6emNu2SZ3U7eDtJoRXd/Zk=; b=RMsrcmW17nKCVZgTeJItmV7/jA41sFpaGROKjkoOf3O86yw6oEJ5e2WjmwKqwU6jDv TnLwEhf7yDdD247BjWfts7sLeCkdy0F5VRXuxBBBFzlGa9quTGaPMgiOPlFvl/y7SZ1F dA5uHkqqC6KG7zzHVcDrl/SLdPeyRGgyuHBKk8/PceKJ07sfnDcwE4t352qeFsCRNUEO gstpYfFCxyE5JqKijElt7yL+TdNg+KJJ6iksa3lkSAzyisD630zSR2uC8A6mP3rbx6+J HFACw7reSVRLOcUTl0NXmnYz0vcuNLuBpt9nxiuUaN/3sPfjLqnL7l3gdg22d8kCoXGe YIbg== X-Forwarded-Encrypted: i=1; AJvYcCWOqDCdQcoUnuUN1L5qSyY+QeyDtR5stQPaH/T7TwkeVCaU6Ne/hX47ufIEDpJTIzXNs8Pm8dYxP2R1uwA=@vger.kernel.org X-Gm-Message-State: AOJu0YzwDf12YJy/wAAt3nIJbMgLuxfNuwugM4e0HB8Ee7uOdOsXcRck fo3/5gThRQxaC31EV8Pb7lI6ckFoYDh41MY0XEKvRH5xJz8ebpOmzVio6VeMpa8= X-Google-Smtp-Source: AGHT+IEoz3UUuA4gFnje/hBBB1Z82zuI7puFcHPYtiCXU5cLSTtZW75S2vlz7SH9FLr/9gF8I3PV6g== X-Received: by 2002:a5d:584b:0:b0:37d:4a2d:6948 with SMTP id ffacd0b85a97d-381c7a5f380mr7629990f8f.33.1730645699563; Sun, 03 Nov 2024 06:54:59 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-381c10d4342sm10785612f8f.32.2024.11.03.06.54.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 06:54:59 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v6 03/13] riscv: Implement cmpxchg32/64() using Zacas Date: Sun, 3 Nov 2024 15:51:43 +0100 Message-Id: <20241103145153.105097-4-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This adds runtime support for Zacas in cmpxchg operations. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones Reviewed-by: Andrea Parri --- arch/riscv/Kconfig | 16 +++++++++++ arch/riscv/Makefile | 3 ++ arch/riscv/include/asm/cmpxchg.h | 48 +++++++++++++++++++++----------- 3 files changed, 50 insertions(+), 17 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 62545946ecf4..3542efe3088b 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -632,6 +632,22 @@ config RISCV_ISA_ZAWRS use of these instructions in the kernel when the Zawrs extension is detected at boot. =20 +config TOOLCHAIN_HAS_ZACAS + bool + default y + depends on !64BIT || $(cc-option,-mabi=3Dlp64 -march=3Drv64ima_zacas) + depends on !32BIT || $(cc-option,-mabi=3Dilp32 -march=3Drv32ima_zacas) + depends on AS_HAS_OPTION_ARCH + +config RISCV_ISA_ZACAS + bool "Zacas extension support for atomic CAS" + depends on TOOLCHAIN_HAS_ZACAS + depends on RISCV_ALTERNATIVE + default y + help + Enable the use of the Zacas ISA-extension to implement kernel atomic + cmpxchg operations when it is detected at boot. + If you don't know what to do here, say Y. =20 config TOOLCHAIN_HAS_ZBB diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index d469db9f46f4..3700a1574413 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -82,6 +82,9 @@ else riscv-march-$(CONFIG_TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI) :=3D $(riscv= -march-y)_zicsr_zifencei endif =20 +# Check if the toolchain supports Zacas +riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZACAS) :=3D $(riscv-march-y)_zacas + # Remove F,D,V from isa string for all. Keep extensions between "fd" and "= v" by # matching non-v and non-multi-letter extensions out with the filter ([^v_= ]*) KBUILD_CFLAGS +=3D -march=3D$(shell echo $(riscv-march-y) | sed -E 's/(rv3= 2ima|rv64ima)fd([^v_]*)v?/\1\2/') diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index ac1d7df898ef..39c1daf39f6a 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -12,6 +12,7 @@ #include #include #include +#include =20 #define __arch_xchg_masked(sc_sfx, prepend, append, r, p, n) \ ({ \ @@ -137,24 +138,37 @@ r =3D (__typeof__(*(p)))((__retx & __mask) >> __s); \ }) =20 -#define __arch_cmpxchg(lr_sfx, sc_sfx, prepend, append, r, p, co, o, n) \ +#define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n= ) \ ({ \ - register unsigned int __rc; \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS)) { \ + r =3D o; \ \ - __asm__ __volatile__ ( \ - prepend \ - "0: lr" lr_sfx " %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc" sc_sfx " %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - append \ - "1:\n" \ - : "=3D&r" (r), "=3D&r" (__rc), "+A" (*(p)) \ - : "rJ" (co o), "rJ" (n) \ - : "memory"); \ + __asm__ __volatile__ ( \ + prepend \ + " amocas" sc_cas_sfx " %0, %z2, %1\n" \ + append \ + : "+&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + } else { \ + register unsigned int __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr" lr_sfx " %0, %2\n" \ + " bne %0, %z3, 1f\n" \ + " sc" sc_cas_sfx " %1, %z4, %2\n" \ + " bnez %1, 0b\n" \ + append \ + "1:\n" \ + : "=3D&r" (r), "=3D&r" (__rc), "+A" (*(p)) \ + : "rJ" (co o), "rJ" (n) \ + : "memory"); \ + } \ }) =20 -#define _arch_cmpxchg(ptr, old, new, sc_sfx, prepend, append) \ +#define _arch_cmpxchg(ptr, old, new, sc_cas_sfx, prepend, append) \ ({ \ __typeof__(ptr) __ptr =3D (ptr); \ __typeof__(*(__ptr)) __old =3D (old); \ @@ -164,15 +178,15 @@ switch (sizeof(*__ptr)) { \ case 1: \ case 2: \ - __arch_cmpxchg_masked(sc_sfx, prepend, append, \ + __arch_cmpxchg_masked(sc_cas_sfx, prepend, append, \ __ret, __ptr, __old, __new); \ break; \ case 4: \ - __arch_cmpxchg(".w", ".w" sc_sfx, prepend, append, \ + __arch_cmpxchg(".w", ".w" sc_cas_sfx, prepend, append, \ __ret, __ptr, (long), __old, __new); \ break; \ case 8: \ - __arch_cmpxchg(".d", ".d" sc_sfx, prepend, append, \ + __arch_cmpxchg(".d", ".d" sc_cas_sfx, prepend, append, \ __ret, __ptr, /**/, __old, __new); \ break; \ default: \ --=20 2.39.2 From nobody Sun Nov 24 17:42:18 2024 Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8C491BE6C for ; Sun, 3 Nov 2024 14:56:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730645764; cv=none; b=eWtPL5D0l5xJsze2tEMCeoe+ivXXtt+uZflU6inRTVpqKUlYdWxue+Uugjo8/FIxPkF+u1FhLqey/ay2ZPhdM6Q0hx9ZO7BdeGeaGvYG/0Bw/SA+LcfASyhUfx0AFvYIFDODUGnzEJ2psvK4c3a8FFthhV+DBrOIBNjxtoC6RH0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730645764; c=relaxed/simple; bh=DRgDwMt2Jlwx1evaYgD0rUR9lyNA/XwrvCTgk+IBusk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=g6Iu31RhdRJcOPOE9ugp8f18psPk0ncNWJa0NS2e+AUI/VC8wA4x37Lojcm3Kt7EIymIPBi0bA/uS6Y+kPuYl7ue1CvnQTmJqTAt3DRK+I5QA1uKwrBw6y6h7QdZtkvbqLFvu1d9vCIQ7MMq0dof9sNn+2Tvfqa9q6O0TUGSvEg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=z6yqiISf; arc=none smtp.client-ip=209.85.128.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="z6yqiISf" Received: by mail-wm1-f42.google.com with SMTP id 5b1f17b1804b1-43155abaf0bso29185815e9.0 for ; Sun, 03 Nov 2024 06:56:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730645761; x=1731250561; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/lAswjGp3xf0DHvJRryPePimKYUd7lF/oN62wU2lB88=; b=z6yqiISfjOngRuB2+EacocBQYut+r7Ta6UjRUXgl+SQuyJWkOyhfMgJNYvBGNGsm2a Ydm7xAv7hq9KP5x1+eeye/B1TRVY2N3Q3u4RlxEI2htd7v2MHlq3nDNvJx9CEaeW/POS Iram1YERIiAdzJXmdzxuC3ZltRhD1kgO6ftfe6G49bK8pVKsy6tomu5pwkthRAlFBvln 2AYtyGBp+9061PIfcLhBtMuNSjH4JArlV/YecWC+PqxFrMrXqLlM2MiaNSlTW+GAnbaM jzT0GJFbXPn5ZmAoOcZ9hh1hRwnIQpeYauP+dAAfqRisAdtVqoo0U8pCGQhjLdYIjWaT /u0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730645761; x=1731250561; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/lAswjGp3xf0DHvJRryPePimKYUd7lF/oN62wU2lB88=; b=huN6EnyrCWZJbQtyPQZ9MZJYGZZG8y4G/7M3EGbAEaHBbCdqscv20ISORIEkIsxRX4 s5rkPIZe+FwKBO3FatumGNwsSc5VvgEhg8fIOws0dPWell2W8B1mr+Vsj9yzqgJejVoW A6Wm8A1fiSRx5MTDn/UAkzGn7MN54XgfFp71ZPeOAaRK2BbIBfmQ6FB3IUV2qt6KOhws tRfhr+zEKPX1xgy1I1fMstpOEmb4oSlzBlRGuyz2H7u2ReLbU71V57fVruRIjqo4ehcY pARsXngN8gGbmJqVYhaUcDhDFvQO++I9xwM6G+fFcADzuEoOseZ4Dkk8ljn/O4pCMTKH oPOQ== X-Forwarded-Encrypted: i=1; AJvYcCU1mIc1eAeZYlO7JSNTiR7aTQWOWYXTZx7c0KssVFiaP+hqWFTwSHm3E+drs5RieWxo0XL/p/Aidb6jxRs=@vger.kernel.org X-Gm-Message-State: AOJu0YyW3puYe7AaA0Q4c7iSyiA5bsj9uEelhyYIske3LOzM/mbtEfO6 qjNbMt0ZqZPNsbpgcJ++k1Huy62XtTgzpmcVO6a09JWffBNzQCcTqOukxZ26g/8= X-Google-Smtp-Source: AGHT+IFXcE4yQOkwclhHZgMus+XU93hkWLBdlkgkjwLbfRe2UgNWPomwyzpyojRGnPIQGSxet5Zozw== X-Received: by 2002:a05:600c:3b04:b0:431:6083:cd38 with SMTP id 5b1f17b1804b1-43283242a52mr72748635e9.6.1730645760870; Sun, 03 Nov 2024 06:56:00 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-381c116ad6fsm10730912f8f.98.2024.11.03.06.55.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 06:56:00 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Conor Dooley , Andrew Jones Subject: [PATCH v6 04/13] dt-bindings: riscv: Add Zabha ISA extension description Date: Sun, 3 Nov 2024 15:51:44 +0100 Message-Id: <20241103145153.105097-5-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add description for the Zabha ISA extension which was ratified in April 2024. Signed-off-by: Alexandre Ghiti Reviewed-by: Guo Ren Acked-by: Conor Dooley Reviewed-by: Andrew Jones Reviewed-by: Andrea Parri --- Documentation/devicetree/bindings/riscv/extensions.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Documentation/devicetree/bindings/riscv/extensions.yaml b/Docu= mentation/devicetree/bindings/riscv/extensions.yaml index 2cf2026cff57..db062107823b 100644 --- a/Documentation/devicetree/bindings/riscv/extensions.yaml +++ b/Documentation/devicetree/bindings/riscv/extensions.yaml @@ -178,6 +178,12 @@ properties: as ratified at commit 4a69197e5617 ("Update to ratified state"= ) of riscv-svvptc. =20 + - const: zabha + description: | + The Zabha extension for Byte and Halfword Atomic Memory Operat= ions + as ratified at commit 49f49c842ff9 ("Update to Rafified state"= ) of + riscv-zabha. + - const: zacas description: | The Zacas extension for Atomic Compare-and-Swap (CAS) instruct= ions --=20 2.39.2 From nobody Sun Nov 24 17:42:18 2024 Received: from mail-wr1-f41.google.com (mail-wr1-f41.google.com [209.85.221.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F010B16DEAC for ; Sun, 3 Nov 2024 14:57:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730645826; cv=none; b=Sm1NgydrQ7YhAUioIu32LNWM+Qb2lIhIOKgdj5H22BKBKsocA5i2LaRUqZUjS1OnB94aZX/RYMnbDtP4N0QjBp20MV+9gXvkI4PFi+voGhx76EEMj5H7wtEBIWTofr1H694Wl2Bu07S1tfYq43RpGSEF5B4E++E2AW1rNxA5MtA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730645826; c=relaxed/simple; bh=87PCTNWJatXkOfxYdKhtjQ3mAAxm4VRDVMIU9aR65aw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=U2dvCrig1oSGmPnBOGPD8s7BgvmyS3Kjx+zt2RoSJWiGGJJb80JlEukd0m3V/XfqHlRYllvFLxNPcQFWkMqRY36Xm9zNLK1c0iskvEnoJu/mqen5j2/bNoEWiJ4yBcnXZUOQX30EmepfMgIzVywyF6fe5eYy73uJq3XXK9xxw5k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=L3Z0HiY2; arc=none smtp.client-ip=209.85.221.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="L3Z0HiY2" Received: by mail-wr1-f41.google.com with SMTP id ffacd0b85a97d-37d6ff1cbe1so2466821f8f.3 for ; Sun, 03 Nov 2024 06:57:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730645822; x=1731250622; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LcjpaetCkzwKl6qLE/ePzZTB5CzyEYuJ8vIVpAvXySA=; b=L3Z0HiY2sXHGtPV6rqYourdnsce2lOJlNFPe4X9SBn8TVni0f2AJbR3F5hekkSo0L3 IngJeCjaKcBNpxZeMaifcIhAfFjROBZYqrj6PQkntp581sN4MWyTCDZqukooxkCdHgAK Bk6PiLOCtnGkwQbNLGcbvfPNhZRfdZxWVOKVxeSGq1tQV//qO2luIxEP4bjAxGRFMHiL QiOg93aLB/3bSKAZY7j+9dCtCJbtwwsURIoW2sC6wtqqEWlDqMnngk2u9WgQ7pVHHFmC WDFCbfsXyLE1veWxQSvdlcPaIn1lKpgacYIgG2eIETwUdo9/82L95KhKypZIOTCLaPh2 QtYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730645822; x=1731250622; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LcjpaetCkzwKl6qLE/ePzZTB5CzyEYuJ8vIVpAvXySA=; b=cu3JXDhJXyZYJjyLiSC21fY680VNJG/CzooJ/01N4XNX2ZCbvspDgAb3kRqfkasPW5 N2sn3aNaTER2NGKcoSgdb2qA98nJl4Cxy2pNli5Yg7rb1IwOoJTJZXwyvB7icz1K9e2Y vNpndOTrUeCglic64BcBVD6qvPj9lUO+hEi/t6hL9empceGhUdQhbZqgC4qcjrZ6q3rQ IfNJyd2NW2DJHuHsyEXMQ3JP4gMolzLalGoSfPV5fKzYs5NQz2B+mX6mkK+ky8z8QFpR 2lFN4Kg+c/kSCJ0cH3p+ItL/BQQXn/QlECs5JM0U2xR+PAvaIqlJaYgPgJl76M8iXvBc 8L5g== X-Forwarded-Encrypted: i=1; AJvYcCWck7+NmSZPl9xW20czTpZrU/p3NDK5x5g22eoj0aljwnuGnQ6+Wq1/B/dEdHiHWYo356gjnpIDZo6xiio=@vger.kernel.org X-Gm-Message-State: AOJu0YyLx1eg5KZuyG6FGvZfmNnvMx3jVp5iC4ztle8ORqDJggM8erGL 0o6dXnN2NsvFc4vV5lw/k3h61tZ1zW7DLcJBSKSK7LAl8hg3YUD56SOL+b6uDa8= X-Google-Smtp-Source: AGHT+IEk9gKHRdbGlFKX3OCkRNc0X43TINOjb8HRMwojR0fr3EGR3rRYNjyJ35dWqeTb53PfklFgKw== X-Received: by 2002:a05:6000:2805:b0:37d:5173:7a54 with SMTP id ffacd0b85a97d-381bea27660mr7207035f8f.52.1730645822142; Sun, 03 Nov 2024 06:57:02 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-381c1168afcsm10825859f8f.91.2024.11.03.06.57.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 06:57:01 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v6 05/13] riscv: Implement cmpxchg8/16() using Zabha Date: Sun, 3 Nov 2024 15:51:45 +0100 Message-Id: <20241103145153.105097-6-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This adds runtime support for Zabha in cmpxchg8/16() operations. Note that in the absence of Zacas support in the toolchain, CAS instructions from Zabha won't be used. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones Reviewed-by: Andrea Parri --- arch/riscv/Kconfig | 18 ++++++++ arch/riscv/Makefile | 3 ++ arch/riscv/include/asm/cmpxchg.h | 78 ++++++++++++++++++++------------ arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/kernel/cpufeature.c | 1 + 5 files changed, 72 insertions(+), 29 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 3542efe3088b..668be90a42e4 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -632,6 +632,24 @@ config RISCV_ISA_ZAWRS use of these instructions in the kernel when the Zawrs extension is detected at boot. =20 +config TOOLCHAIN_HAS_ZABHA + bool + default y + depends on !64BIT || $(cc-option,-mabi=3Dlp64 -march=3Drv64ima_zabha) + depends on !32BIT || $(cc-option,-mabi=3Dilp32 -march=3Drv32ima_zabha) + depends on AS_HAS_OPTION_ARCH + +config RISCV_ISA_ZABHA + bool "Zabha extension support for atomic byte/halfword operations" + depends on TOOLCHAIN_HAS_ZABHA + depends on RISCV_ALTERNATIVE + default y + help + Enable the use of the Zabha ISA-extension to implement kernel + byte/halfword atomic memory operations when it is detected at boot. + + If you don't know what to do here, say Y. + config TOOLCHAIN_HAS_ZACAS bool default y diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index 3700a1574413..9fe1ee740dda 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -85,6 +85,9 @@ endif # Check if the toolchain supports Zacas riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZACAS) :=3D $(riscv-march-y)_zacas =20 +# Check if the toolchain supports Zabha +riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZABHA) :=3D $(riscv-march-y)_zabha + # Remove F,D,V from isa string for all. Keep extensions between "fd" and "= v" by # matching non-v and non-multi-letter extensions out with the filter ([^v_= ]*) KBUILD_CFLAGS +=3D -march=3D$(shell echo $(riscv-march-y) | sed -E 's/(rv3= 2ima|rv64ima)fd([^v_]*)v?/\1\2/') diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index 39c1daf39f6a..1f4cd12e4664 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -108,34 +108,49 @@ * indicated by comparing RETURN with OLD. */ =20 -#define __arch_cmpxchg_masked(sc_sfx, prepend, append, r, p, o, n) \ -({ \ - u32 *__ptr32b =3D (u32 *)((ulong)(p) & ~0x3); \ - ulong __s =3D ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ - ulong __mask =3D GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ - << __s; \ - ulong __newx =3D (ulong)(n) << __s; \ - ulong __oldx =3D (ulong)(o) << __s; \ - ulong __retx; \ - ulong __rc; \ - \ - __asm__ __volatile__ ( \ - prepend \ - "0: lr.w %0, %2\n" \ - " and %1, %0, %z5\n" \ - " bne %1, %z3, 1f\n" \ - " and %1, %0, %z6\n" \ - " or %1, %1, %z4\n" \ - " sc.w" sc_sfx " %1, %1, %2\n" \ - " bnez %1, 0b\n" \ - append \ - "1:\n" \ - : "=3D&r" (__retx), "=3D&r" (__rc), "+A" (*(__ptr32b)) \ - : "rJ" ((long)__oldx), "rJ" (__newx), \ - "rJ" (__mask), "rJ" (~__mask) \ - : "memory"); \ - \ - r =3D (__typeof__(*(p)))((__retx & __mask) >> __s); \ +#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, prepend, append, r, p, o, n= ) \ +({ \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && \ + IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZABHA) && \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS)) { \ + r =3D o; \ + \ + __asm__ __volatile__ ( \ + prepend \ + " amocas" cas_sfx " %0, %z2, %1\n" \ + append \ + : "+&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + } else { \ + u32 *__ptr32b =3D (u32 *)((ulong)(p) & ~0x3); \ + ulong __s =3D ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ + ulong __mask =3D GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ + << __s; \ + ulong __newx =3D (ulong)(n) << __s; \ + ulong __oldx =3D (ulong)(o) << __s; \ + ulong __retx; \ + ulong __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr.w %0, %2\n" \ + " and %1, %0, %z5\n" \ + " bne %1, %z3, 1f\n" \ + " and %1, %0, %z6\n" \ + " or %1, %1, %z4\n" \ + " sc.w" sc_sfx " %1, %1, %2\n" \ + " bnez %1, 0b\n" \ + append \ + "1:\n" \ + : "=3D&r" (__retx), "=3D&r" (__rc), "+A" (*(__ptr32b)) \ + : "rJ" ((long)__oldx), "rJ" (__newx), \ + "rJ" (__mask), "rJ" (~__mask) \ + : "memory"); \ + \ + r =3D (__typeof__(*(p)))((__retx & __mask) >> __s); \ + } \ }) =20 #define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n= ) \ @@ -177,8 +192,13 @@ \ switch (sizeof(*__ptr)) { \ case 1: \ + __arch_cmpxchg_masked(sc_cas_sfx, ".b" sc_cas_sfx, \ + prepend, append, \ + __ret, __ptr, __old, __new); \ + break; \ case 2: \ - __arch_cmpxchg_masked(sc_cas_sfx, prepend, append, \ + __arch_cmpxchg_masked(sc_cas_sfx, ".h" sc_cas_sfx, \ + prepend, append, \ __ret, __ptr, __old, __new); \ break; \ case 4: \ diff --git a/arch/riscv/include/asm/hwcap.h b/arch/riscv/include/asm/hwcap.h index 46d9de54179e..74bcb0e2bd1f 100644 --- a/arch/riscv/include/asm/hwcap.h +++ b/arch/riscv/include/asm/hwcap.h @@ -93,6 +93,7 @@ #define RISCV_ISA_EXT_ZCMOP 84 #define RISCV_ISA_EXT_ZAWRS 85 #define RISCV_ISA_EXT_SVVPTC 86 +#define RISCV_ISA_EXT_ZABHA 87 =20 #define RISCV_ISA_EXT_XLINUXENVCFG 127 =20 diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index 3a8eeaa9310c..5e743d8d34f5 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -322,6 +322,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] =3D { __RISCV_ISA_EXT_DATA(zihintpause, RISCV_ISA_EXT_ZIHINTPAUSE), __RISCV_ISA_EXT_DATA(zihpm, RISCV_ISA_EXT_ZIHPM), __RISCV_ISA_EXT_DATA(zimop, RISCV_ISA_EXT_ZIMOP), + __RISCV_ISA_EXT_DATA(zabha, RISCV_ISA_EXT_ZABHA), __RISCV_ISA_EXT_DATA(zacas, RISCV_ISA_EXT_ZACAS), __RISCV_ISA_EXT_DATA(zawrs, RISCV_ISA_EXT_ZAWRS), __RISCV_ISA_EXT_DATA(zfa, RISCV_ISA_EXT_ZFA), --=20 2.39.2 From nobody Sun Nov 24 17:42:18 2024 Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 913DB170A16 for ; Sun, 3 Nov 2024 14:58:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730645888; cv=none; b=UvhXvE3LPqcO5k9QIeOmr2eK0rP0VwuEwGSGMo1oM06BWMTeFKeIYET5qlW3lxy30k9fheAArVVhi2sQ/YxHl+ceYs7zwSruFeo08BcAkLXe7SPEQ/09zLTAss5fP8fhN0l78BbctMU2nacYCowj9VhSgLHyOD0ew/syRtKIBU0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730645888; c=relaxed/simple; bh=AA0WiotsCMovaMp28+X0tg8CsoVA5Lev8Q+VorcC64k=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rnkuQzKdjqAtLCkl32EyvsgP9vGT7IklzNMnfLMYg1F4d375x1Fiw78syH1dZ1UkmCEtyTepHW9F77unSgbc9o2lqJZnR970pHnZWk6TlBuA1qjfr66swIfjpBwAztiiVMIMzSm0WRm02V9w82ZbjL/pmVHSBPxm7Q7AfVASDTs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=SUV48zvU; arc=none smtp.client-ip=209.85.128.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="SUV48zvU" Received: by mail-wm1-f51.google.com with SMTP id 5b1f17b1804b1-4315c1c7392so29105505e9.1 for ; Sun, 03 Nov 2024 06:58:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730645884; x=1731250684; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=N5AV+nOoC/tuklQRoqzKvrTd6uHJ4OaRW4xZaKd3wLE=; b=SUV48zvUYIzHcO2HQf0r+ojNeM5Vbt38AQAgwNt5+y3wrUVCYPiRGNk6RV3NxYc7gn XzILqq/JFUOyNt30Nm2ZLXCLLTrfE6Z665EN8ju9KWVnBGRDlmMbKv//knBCj53pWLx+ lpNO8qXCeS2yKHxo9FGf45lI2wrxRwRoOTOhLnT+CQqV9F9lpQF2foNwXaPML8zn7+q8 yfQLQYm4DPBKEckk7SAPaYlrZvltWcjjpDtApEsed1HZxcniheT1xBBH9eQQqG7E9NcG IN51SUm776Vuar/B1kJ7sz46vPKNQql5qWfLU5IiBjedKnLBMvwvg9IlhmscFAbhPh9F 1rVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730645884; x=1731250684; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=N5AV+nOoC/tuklQRoqzKvrTd6uHJ4OaRW4xZaKd3wLE=; b=ev6w7H+FkdcbkGSu+JUNW6v5EnCt+cRqTgiyNlku5Uw02SyjxUAfoU2XO7//IazVCm sRtO2YU1+YE3SIrmyP2h/t7EQhpAQlc+MKw7tHeZwPKtuRwb5dHFyqk9xQw7c0hFzdlM ywKPfXdZIBLMKqj4Nc0Ig0KcCL/ikzh+gW7FAGOJguaQY0xbx+VdFgA3vH62H9AwOmOq X43KYPy2oAwxx0DDR0kX7KDOI5LpuqYDOf6TBJzhLdL6iv/SbBnD54RLT10PLGxw7OnJ keA6oMNM1Mb3vsftEETVYuhtXn7lp4MxoaB8XNjNMvKr1I1JzfKFDYD4SjSDoPIxxQ0Q t7kQ== X-Forwarded-Encrypted: i=1; AJvYcCVKcaV4C3JQPk02gdbyngAKXqOrfTorunqF/TmLgEcvpFi20YMs0CK/kgPwLc56izN6vhrtEhFX7xYac3I=@vger.kernel.org X-Gm-Message-State: AOJu0YxTkyfcF6769szgK8B8RppH3zLdlIvW6OqocRtnEf1+1JIQ6t4u 7N14RG/Xi6ySOIjETb5vUMKZiJ1fqXdxkZaM+RHk5uLyJpHd+1D12TucaeTus6w= X-Google-Smtp-Source: AGHT+IE55nBQ7rbTlkCqdHR3oJ0XymsMVwn2Z7Lo05//Ad0W+sZxb56nu4edxwe80UlQVGkf+GZkPg== X-Received: by 2002:a05:600c:22d3:b0:432:7c08:d121 with SMTP id 5b1f17b1804b1-432849fa03amr81532075e9.12.1730645883736; Sun, 03 Nov 2024 06:58:03 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4327d5c65b8sm124358355e9.18.2024.11.03.06.58.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 06:58:03 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrea Parri , Andrew Jones Subject: [PATCH v6 06/13] riscv: Improve zacas fully-ordered cmpxchg() Date: Sun, 3 Nov 2024 15:51:46 +0100 Message-Id: <20241103145153.105097-7-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The current fully-ordered cmpxchgXX() implementation results in: amocas.X.rl a5,a4,(s1) fence rw,rw This provides enough sync but we can actually use the following better mapping instead: amocas.X.aqrl a5,a4,(s1) Suggested-by: Andrea Parri Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones Reviewed-by: Andrea Parri --- arch/riscv/include/asm/cmpxchg.h | 92 ++++++++++++++++++++++---------- 1 file changed, 64 insertions(+), 28 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index 1f4cd12e4664..052418aba11a 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -107,8 +107,10 @@ * store NEW in MEM. Return the initial value in MEM. Success is * indicated by comparing RETURN with OLD. */ - -#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, prepend, append, r, p, o, n= ) \ +#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + r, p, o, n) \ ({ \ if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && \ IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \ @@ -117,9 +119,9 @@ r =3D o; \ \ __asm__ __volatile__ ( \ - prepend \ + cas_prepend \ " amocas" cas_sfx " %0, %z2, %1\n" \ - append \ + cas_append \ : "+&r" (r), "+A" (*(p)) \ : "rJ" (n) \ : "memory"); \ @@ -134,7 +136,7 @@ ulong __rc; \ \ __asm__ __volatile__ ( \ - prepend \ + sc_prepend \ "0: lr.w %0, %2\n" \ " and %1, %0, %z5\n" \ " bne %1, %z3, 1f\n" \ @@ -142,7 +144,7 @@ " or %1, %1, %z4\n" \ " sc.w" sc_sfx " %1, %1, %2\n" \ " bnez %1, 0b\n" \ - append \ + sc_append \ "1:\n" \ : "=3D&r" (__retx), "=3D&r" (__rc), "+A" (*(__ptr32b)) \ : "rJ" ((long)__oldx), "rJ" (__newx), \ @@ -153,16 +155,19 @@ } \ }) =20 -#define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n= ) \ +#define __arch_cmpxchg(lr_sfx, sc_sfx, cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + r, p, co, o, n) \ ({ \ if (IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \ riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS)) { \ r =3D o; \ \ __asm__ __volatile__ ( \ - prepend \ - " amocas" sc_cas_sfx " %0, %z2, %1\n" \ - append \ + cas_prepend \ + " amocas" cas_sfx " %0, %z2, %1\n" \ + cas_append \ : "+&r" (r), "+A" (*(p)) \ : "rJ" (n) \ : "memory"); \ @@ -170,12 +175,12 @@ register unsigned int __rc; \ \ __asm__ __volatile__ ( \ - prepend \ + sc_prepend \ "0: lr" lr_sfx " %0, %2\n" \ " bne %0, %z3, 1f\n" \ - " sc" sc_cas_sfx " %1, %z4, %2\n" \ + " sc" sc_sfx " %1, %z4, %2\n" \ " bnez %1, 0b\n" \ - append \ + sc_append \ "1:\n" \ : "=3D&r" (r), "=3D&r" (__rc), "+A" (*(p)) \ : "rJ" (co o), "rJ" (n) \ @@ -183,7 +188,9 @@ } \ }) =20 -#define _arch_cmpxchg(ptr, old, new, sc_cas_sfx, prepend, append) \ +#define _arch_cmpxchg(ptr, old, new, sc_sfx, cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append) \ ({ \ __typeof__(ptr) __ptr =3D (ptr); \ __typeof__(*(__ptr)) __old =3D (old); \ @@ -192,22 +199,28 @@ \ switch (sizeof(*__ptr)) { \ case 1: \ - __arch_cmpxchg_masked(sc_cas_sfx, ".b" sc_cas_sfx, \ - prepend, append, \ - __ret, __ptr, __old, __new); \ + __arch_cmpxchg_masked(sc_sfx, ".b" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, __old, __new); \ break; \ case 2: \ - __arch_cmpxchg_masked(sc_cas_sfx, ".h" sc_cas_sfx, \ - prepend, append, \ - __ret, __ptr, __old, __new); \ + __arch_cmpxchg_masked(sc_sfx, ".h" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, __old, __new); \ break; \ case 4: \ - __arch_cmpxchg(".w", ".w" sc_cas_sfx, prepend, append, \ - __ret, __ptr, (long), __old, __new); \ + __arch_cmpxchg(".w", ".w" sc_sfx, ".w" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, (long), __old, __new); \ break; \ case 8: \ - __arch_cmpxchg(".d", ".d" sc_cas_sfx, prepend, append, \ - __ret, __ptr, /**/, __old, __new); \ + __arch_cmpxchg(".d", ".d" sc_sfx, ".d" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, /**/, __old, __new); \ break; \ default: \ BUILD_BUG(); \ @@ -215,17 +228,40 @@ (__typeof__(*(__ptr)))__ret; \ }) =20 +/* + * These macros are here to improve the readability of the arch_cmpxchg_XX= X() + * macros. + */ +#define SC_SFX(x) x +#define CAS_SFX(x) x +#define SC_PREPEND(x) x +#define SC_APPEND(x) x +#define CAS_PREPEND(x) x +#define CAS_APPEND(x) x + #define arch_cmpxchg_relaxed(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", "", "") + _arch_cmpxchg((ptr), (o), (n), \ + SC_SFX(""), CAS_SFX(""), \ + SC_PREPEND(""), SC_APPEND(""), \ + CAS_PREPEND(""), CAS_APPEND("")) =20 #define arch_cmpxchg_acquire(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", "", RISCV_ACQUIRE_BARRIER) + _arch_cmpxchg((ptr), (o), (n), \ + SC_SFX(""), CAS_SFX(""), \ + SC_PREPEND(""), SC_APPEND(RISCV_ACQUIRE_BARRIER), \ + CAS_PREPEND(""), CAS_APPEND(RISCV_ACQUIRE_BARRIER)) =20 #define arch_cmpxchg_release(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", RISCV_RELEASE_BARRIER, "") + _arch_cmpxchg((ptr), (o), (n), \ + SC_SFX(""), CAS_SFX(""), \ + SC_PREPEND(RISCV_RELEASE_BARRIER), SC_APPEND(""), \ + CAS_PREPEND(RISCV_RELEASE_BARRIER), CAS_APPEND("")) =20 #define arch_cmpxchg(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), ".rl", "", " fence rw, rw\n") + _arch_cmpxchg((ptr), (o), (n), \ + SC_SFX(".rl"), CAS_SFX(".aqrl"), \ + SC_PREPEND(""), SC_APPEND(RISCV_FULL_BARRIER), \ + CAS_PREPEND(""), CAS_APPEND("")) =20 #define arch_cmpxchg_local(ptr, o, n) \ arch_cmpxchg_relaxed((ptr), (o), (n)) --=20 2.39.2 From nobody Sun Nov 24 17:42:18 2024 Received: from mail-wr1-f50.google.com (mail-wr1-f50.google.com [209.85.221.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BE9161714A8 for ; Sun, 3 Nov 2024 14:59:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730645948; cv=none; b=NAuZEmIJdXL3TrWlJ8L3NTwD/swHzwv/pXQ+Edc+W18s52HzfYIg+BWwZjp79wkFi2Mx/gzjRrKqw8dz72M08rWJo9gsvM1wRl/cOSqHCVG73RPGZpcYhMW+/XS7o9EQQUHJffyrvVDG7lT0Yg4+B/DzkqHfxjHt5Jz/L4kTTC8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730645948; c=relaxed/simple; bh=owSN92DfUo1jaGZz+2Ukq4uBNlzp0f+9tLQB0G3n0sE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ZKygVXIwmsytwVLEP5vjib61gwDoHUHs6b+H8EzAnu5Sq8TP6rLmQzpdPqavCkGDFp0lgqqZdUUKBbUCjbB32UMI6/idLPdIOcelYz69p1OPjAdM1q8pbQoFTL3o99qxUFbaWdFhBr7/tNJ5X0wWctw5PlnTh00U5CxAoul7XKA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=W9s4dmif; arc=none smtp.client-ip=209.85.221.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="W9s4dmif" Received: by mail-wr1-f50.google.com with SMTP id ffacd0b85a97d-37d8901cb98so2655001f8f.0 for ; Sun, 03 Nov 2024 06:59:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730645945; x=1731250745; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2YMddV/TixlRcMkBqzVFy3RqDe67DeCntBmj39JunYc=; b=W9s4dmifzYl8kc3GW3buR4m0AoaYA23sXD6nTU/09kuDlJXP/EA2Y6yVCffn9aE44v 8esJryubesuCvNO58QQmuS3C8wRkWyPYMzXwwANy5vQCFSG1gqbxzLzYVwBxhpy3DI6y eBHGnu8Q37yWK/4GAdyB7HpdwapbRGEsEnG1Hb0LcrVpfKtp/vEj9eNzmR7xajmhnhs9 aN9jFYESNGYXjQpd3vGuduhOogDweWqI2Ce8jHBUiRb6hSC0DQ/AwQiumgiQ/pN7coMN DaCoadJlgTyql7cU0IcqZQmBF8LMNk68nDc7L9jbbEnSAB+tt20cGMnAg0LZHU2+XMN3 Yw9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730645945; x=1731250745; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2YMddV/TixlRcMkBqzVFy3RqDe67DeCntBmj39JunYc=; b=a8yrgSaI/6Yy965xPn9JGj0UNeCgPdVsKz/+hMX+35R984Xh8tjrxAPr9l00hSZrB+ 3onTTzIWnNq48sjGLJMtLq80d+WYykTrI/JFfB7x68Zp2zVMqSi8Vao+fb6PlIsXR3n1 KKz0jOS5uLoPSYCD1JEg0rLU707d2yhAa0x/6edzyxZtLngYoC+J5KobqSPLRpIsvKv3 pPvD30WuhQYGHNdZx+3Gm7aK3ZtawCenA1dAITlMotyjBnwSGylAktDrIWga3vdDsb/B lBGPceqHOUsct0v+fKUcWMoKIngqTxkiArZ8boui695oTTXmK7X6aEZCDOTlsOeJv67Q 5Vzw== X-Forwarded-Encrypted: i=1; AJvYcCX+sHlkzETQytEsehNaX9SgtqUb7DTUVWQ3c8RoTQXPnOx2gFX5ehd2lzvn5V5GNre079glgIDa8/HrLoc=@vger.kernel.org X-Gm-Message-State: AOJu0Ywo7ntER8EKiBhHZ/XfVuyJk+vFykvYEb47bZMo1CHJlFeBWzNn xn2iDhOel8lvy0JZzlBXCOCNIMpcO+HTbQME7XXJo0QCljKPThzX7wB4ynCo5pI= X-Google-Smtp-Source: AGHT+IEwAhzLmFndg4dtG6o7eex1DY00zUsKhGi9VFC31IF8dEpvf7rYFDq+11tOCJeycg1G7RqbUQ== X-Received: by 2002:a5d:5a8c:0:b0:381:d133:d541 with SMTP id ffacd0b85a97d-381d133d9cdmr3234403f8f.11.1730645945038; Sun, 03 Nov 2024 06:59:05 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4327d6852d9sm126046165e9.37.2024.11.03.06.59.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 06:59:04 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v6 07/13] riscv: Implement arch_cmpxchg128() using Zacas Date: Sun, 3 Nov 2024 15:51:47 +0100 Message-Id: <20241103145153.105097-8-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that Zacas is supported in the kernel, let's use the double word atomic version of amocas to improve the SLUB allocator. Note that we have to select fixed registers, otherwise gcc fails to pick even registers and then produces a reserved encoding which fails to assemble. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones Reviewed-by: Andrea Parri --- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/cmpxchg.h | 38 ++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 668be90a42e4..093ee6537331 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -115,6 +115,7 @@ config RISCV select GENERIC_VDSO_TIME_NS if HAVE_GENERIC_VDSO select HARDIRQS_SW_RESEND select HAS_IOPORT if MMU + select HAVE_ALIGNED_STRUCT_PAGE select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMALLOC if HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_HUGE_VMAP if MMU && 64BIT diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index 052418aba11a..f95929f538b2 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -296,6 +296,44 @@ arch_cmpxchg_release((ptr), (o), (n)); \ }) =20 +#if defined(CONFIG_64BIT) && defined(CONFIG_RISCV_ISA_ZACAS) + +#define system_has_cmpxchg128() riscv_has_extension_unlikely(RISCV_= ISA_EXT_ZACAS) + +union __u128_halves { + u128 full; + struct { + u64 low, high; + }; +}; + +#define __arch_cmpxchg128(p, o, n, cas_sfx) \ +({ \ + __typeof__(*(p)) __o =3D (o); = \ + union __u128_halves __hn =3D { .full =3D (n) }; \ + union __u128_halves __ho =3D { .full =3D (__o) }; \ + register unsigned long t1 asm ("t1") =3D __hn.low; \ + register unsigned long t2 asm ("t2") =3D __hn.high; \ + register unsigned long t3 asm ("t3") =3D __ho.low; \ + register unsigned long t4 asm ("t4") =3D __ho.high; \ + \ + __asm__ __volatile__ ( \ + " amocas.q" cas_sfx " %0, %z3, %2" \ + : "+&r" (t3), "+&r" (t4), "+A" (*(p)) \ + : "rJ" (t1), "rJ" (t2) \ + : "memory"); \ + \ + ((u128)t4 << 64) | t3; \ +}) + +#define arch_cmpxchg128(ptr, o, n) \ + __arch_cmpxchg128((ptr), (o), (n), ".aqrl") + +#define arch_cmpxchg128_local(ptr, o, n) \ + __arch_cmpxchg128((ptr), (o), (n), "") + +#endif /* CONFIG_64BIT && CONFIG_RISCV_ISA_ZACAS */ + #ifdef CONFIG_RISCV_ISA_ZAWRS /* * Despite wrs.nto being "WRS-with-no-timeout", in the absence of changes = to --=20 2.39.2 From nobody Sun Nov 24 17:42:18 2024 Received: from mail-wr1-f49.google.com (mail-wr1-f49.google.com [209.85.221.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2E4AB17BB16 for ; Sun, 3 Nov 2024 15:00:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730646011; cv=none; b=obd2rFp+Mkznnxp+BSaLd38YzTgCB73A0cTpiXVnHq5G5chN031jqqvPkgGmaSTtWSTrwJ3JuvOHtIoCkyhgxxQsM9BksPfjkZ96LAshWBLRg4zMwu+GQbVBSjvSV4+eBTqvKIRTOKhN0SrbHIxRP6IfzwhWqpCjynmXvnEU9tQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730646011; c=relaxed/simple; bh=Md27kdouLX9lf0nN1/O929Tij6j8YRbNusJGqq+3+5U=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ol/NdFkMEu80CKHhS1zoiVLEl+d3UcLVzFFAaExBAitGvC3Qjc+s0IzT2+pJtYHXhKFLzyJ149HcdEUe5V6ffD7JY5CPzG+JxIXVc/2v/fa6MXjowA/+i4fCP3Mv4kWcpZ9WbmOqppuGFc5vGP0LC8GhhKlfNfhgZF8WWWEzemY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=T7baZdCR; arc=none smtp.client-ip=209.85.221.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="T7baZdCR" Received: by mail-wr1-f49.google.com with SMTP id ffacd0b85a97d-37ec4e349f4so2144578f8f.0 for ; Sun, 03 Nov 2024 07:00:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730646006; x=1731250806; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=C7omRwb2buD8uK45bQUSORyOPEIU9zvIfXDatLe8IP8=; b=T7baZdCR99cg09+6AmG//hj1PZFk16ugIx52lFcIwW02YrQHUKkB0llKNkL/+CTEQ3 0x2Pwr8E6rGybSY3efHR3tkDrM+/tWzLn/NPedq5TPREguTcNWCgcRYgdeZrVPH3GOrB 36UtvdoGh9IhWnsuVpf13XwuWbE7/8Co5RRw3AcS4TYL8e4uV7vLbuxziHkAunkaLC9Y +Dv1JJ6d1L36M/Bjl7GGF4mMrFgvEkN1IHxqPubKb/NyTKYjOs9uNQxSSzpSuf32zIEI Vq42xGSZGnmM+BkrQTrgYYHpJ2Jxkf6UAhdoRODsUc91hYc1CEdl7RafzLlXR7Vm68nK pJSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730646006; x=1731250806; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=C7omRwb2buD8uK45bQUSORyOPEIU9zvIfXDatLe8IP8=; b=QkMkCCenzWg8pkdTOHDZGaeXpPHAcK488MlksHLUBuySgADmVfZE20Q0BapZLq5T4c 0X/bEMtSSd5Li+h5Fv1AROYMN+kQDHBNXU2fCOpJQ9+APlWXODxmkbm3lPH+1Jx8ANxB UtEhHqFjUHzp48j9Bx8JdjN+k+sOLKO3MZBbe5C/mvUI8S+GJ0gVG1kiDc/XnOQJYlTt dJBF1uFc/zJ25JZNh/C75lzWi6CFpua7t92THjRjk6SWHMrAB1+0+JXAavWvz0wFMVz+ GfHQyI3dRh01USHkLSkNKfFL8FGe0/tFcouLKTEIQ8FqItLOyr+TmHV3kbNkfDba7i4f fhEA== X-Forwarded-Encrypted: i=1; AJvYcCVgF1cLzcN36E63r2cWsIUrWbPFilJBB2kcDDOcxfoC6+hPHDIKaI1fSca3O3xwy40IJ5qr+PX2h16k9ig=@vger.kernel.org X-Gm-Message-State: AOJu0YyW8933h0O/KnRAY32fZrDnUH9F+pQDO/U6pa9vlamJRGS4Q3Fn ky2hHPW3u7vAwhvaylu1+bonLseKpEsCf7T2bDEUaxGaKxiK7jPGBbOJjxNFIhk= X-Google-Smtp-Source: AGHT+IErbU6/+ESL9c7D4Rkx4JGXztbjh558spmD2njQc7VJwfVBMIIjkXAoMafrlIu7KWoRdehU9Q== X-Received: by 2002:a05:6000:4029:b0:37e:d2b7:acd5 with SMTP id ffacd0b85a97d-381c7a476a8mr7741924f8f.8.1730646006376; Sun, 03 Nov 2024 07:00:06 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-381c10e7465sm10805300f8f.53.2024.11.03.07.00.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 07:00:05 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v6 08/13] riscv: Implement xchg8/16() using Zabha Date: Sun, 3 Nov 2024 15:51:48 +0100 Message-Id: <20241103145153.105097-9-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This adds runtime support for Zabha in xchg8/16() operations. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones Reviewed-by: Andrea Parri --- arch/riscv/include/asm/cmpxchg.h | 65 ++++++++++++++++++++------------ 1 file changed, 41 insertions(+), 24 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index f95929f538b2..4cadc56220fe 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -14,29 +14,41 @@ #include #include =20 -#define __arch_xchg_masked(sc_sfx, prepend, append, r, p, n) \ -({ \ - u32 *__ptr32b =3D (u32 *)((ulong)(p) & ~0x3); \ - ulong __s =3D ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ - ulong __mask =3D GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ - << __s; \ - ulong __newx =3D (ulong)(n) << __s; \ - ulong __retx; \ - ulong __rc; \ - \ - __asm__ __volatile__ ( \ - prepend \ - "0: lr.w %0, %2\n" \ - " and %1, %0, %z4\n" \ - " or %1, %1, %z3\n" \ - " sc.w" sc_sfx " %1, %1, %2\n" \ - " bnez %1, 0b\n" \ - append \ - : "=3D&r" (__retx), "=3D&r" (__rc), "+A" (*(__ptr32b)) \ - : "rJ" (__newx), "rJ" (~__mask) \ - : "memory"); \ - \ - r =3D (__typeof__(*(p)))((__retx & __mask) >> __s); \ +#define __arch_xchg_masked(sc_sfx, swap_sfx, prepend, sc_append, \ + swap_append, r, p, n) \ +({ \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZABHA)) { \ + __asm__ __volatile__ ( \ + prepend \ + " amoswap" swap_sfx " %0, %z2, %1\n" \ + swap_append \ + : "=3D&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + } else { \ + u32 *__ptr32b =3D (u32 *)((ulong)(p) & ~0x3); \ + ulong __s =3D ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ + ulong __mask =3D GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ + << __s; \ + ulong __newx =3D (ulong)(n) << __s; \ + ulong __retx; \ + ulong __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr.w %0, %2\n" \ + " and %1, %0, %z4\n" \ + " or %1, %1, %z3\n" \ + " sc.w" sc_sfx " %1, %1, %2\n" \ + " bnez %1, 0b\n" \ + sc_append \ + : "=3D&r" (__retx), "=3D&r" (__rc), "+A" (*(__ptr32b)) \ + : "rJ" (__newx), "rJ" (~__mask) \ + : "memory"); \ + \ + r =3D (__typeof__(*(p)))((__retx & __mask) >> __s); \ + } \ }) =20 #define __arch_xchg(sfx, prepend, append, r, p, n) \ @@ -59,8 +71,13 @@ \ switch (sizeof(*__ptr)) { \ case 1: \ + __arch_xchg_masked(sc_sfx, ".b" swap_sfx, \ + prepend, sc_append, swap_append, \ + __ret, __ptr, __new); \ + break; \ case 2: \ - __arch_xchg_masked(sc_sfx, prepend, sc_append, \ + __arch_xchg_masked(sc_sfx, ".h" swap_sfx, \ + prepend, sc_append, swap_append, \ __ret, __ptr, __new); \ break; \ case 4: \ --=20 2.39.2 From nobody Sun Nov 24 17:42:18 2024 Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C1476158522 for ; Sun, 3 Nov 2024 15:01:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730646071; cv=none; b=U58Tc4w42hBqsDlq8pVUPySfCpkAAp0eiNc87migNr0MpnVrV9pqK6cURDsYrsz0WyyvverpBVGtof5HQ0uD+tQV8hidJfa7RsSPgcmqmE9TM90rKOlz8VE/jvcZs/yARTG2fx/I+0Cdl3Yd2S5kdts16OI4HECVKNOh3vpz8gc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730646071; c=relaxed/simple; bh=EDwD9aZEGDLa1mlnyZaiQGHjsh7Aq5OXTHhH9BR6X5M=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=TBjeVRrcykzHdrjgEHEACjozyuGLXuL1nDEBjxeIoJCUFChQ8AHGTzXSQg5Fu3Ieo6pRLirTdFnuekWYgGjsBEKy9o+IVzUNxNq/oUtez4KgIUYmaFFJBkWJ9VpF5IlgNqQEbp7LRDoI6nu68tMgHfs6Lq+rNnl0QyBLy7fn5uE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=qGkT5xhP; arc=none smtp.client-ip=209.85.128.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="qGkT5xhP" Received: by mail-wm1-f46.google.com with SMTP id 5b1f17b1804b1-43169902057so26372375e9.0 for ; Sun, 03 Nov 2024 07:01:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730646068; x=1731250868; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2s2g6PiZXcRkI3A8BFgYC9wzyXLIo5JanvDPO+lSWQk=; b=qGkT5xhPAMhRMeXBOW/1eKsYXMmqh9iyK2rw8zNy/oYDmZMiTsTm7r+7SLbCooKgew wqorE3cNqAingkKpq85t78fKSjYeny2W62SnhJRp2HrBicOz5rwDUv9vIS44/24nlYfn nSJNoixhHWxagg76s1tf+K5SzbOEsGGESQqwKdoZN40O/WwOso3oa6tWqCwr1oYuQTJ2 Xtj+ff6KOtNS66BtcFjmn1LkU3NZzzEsaryq0C5Kbnv86zgmqK9sHIAl/PMvVhuxfUZU wZdPJDM4XUe9Yr9k7yCW7m+x7oZYrgj05+wc5huths1BDViYuHEAdffIgtLEWH4xlvRn tsXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730646068; x=1731250868; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2s2g6PiZXcRkI3A8BFgYC9wzyXLIo5JanvDPO+lSWQk=; b=DWQFZ8Jt3n2aA2eJlsjElTMpfYZS0Ix+LG/HDwoeatKFWOHFDXr+AgHLsuTsrZLUEl cHT4jHkWqlnritKkKLfBEp+B1FLIdJ7cYNuRnFJLNil3MdDaROH7IsxCzja8m4ozByDw kxy+eLEQzmWl/HEqrJnWiQz07NkKcue+krbZWUnVeApP/nEzI1IHl72b6ln3JnhoZYHQ qITfK0w81bD7Sl2akNuNs4QROtVSrvl5z5bOtr3+l1OnDeVelAGfwVbGroJyd49/ymvK 2E+WM2g9GVeP5byUmNTHCbN9dQcqZp6FVwWgzthtiIJLUe0M2mNrAhgtyV5/JQmfewLO bIqQ== X-Forwarded-Encrypted: i=1; AJvYcCUdNg/mqicAElKa8a1Z4WUsMAkTYq73TqYqieM7LsPZKYbmOZw1c6igzTn127nQNdg3V4NjqLTKM6O6zjQ=@vger.kernel.org X-Gm-Message-State: AOJu0Yz9v/fvLftRp7rNPhbXRMcMaeGgvRIHigxLYxf5AfdYaDAsesPm 205r8oHQ/WvZWvy/HVe9ZW7jJ3nxY1oIMw95AOs923Frp7snphn6CcDDVDFVhKE= X-Google-Smtp-Source: AGHT+IHT05faWmtqnYuiNBzL9haD0/7Dkr0QGrJhaTkS4osZpzeT6/FO8hvYgpCC2wE7TfnKCk1UVw== X-Received: by 2002:a05:600c:4215:b0:431:55af:a230 with SMTP id 5b1f17b1804b1-431bb9e6031mr145075625e9.33.1730646067731; Sun, 03 Nov 2024 07:01:07 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4327d6852fdsm125977405e9.34.2024.11.03.07.01.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 07:01:07 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Guo Ren , Andrew Jones Subject: [PATCH v6 09/13] asm-generic: ticket-lock: Reuse arch_spinlock_t of qspinlock Date: Sun, 3 Nov 2024 15:51:49 +0100 Message-Id: <20241103145153.105097-10-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Guo Ren The arch_spinlock_t of qspinlock has contained the atomic_t val, which satisfies the ticket-lock requirement. Thus, unify the arch_spinlock_t into qspinlock_types.h. This is the preparation for the next combo spinlock. Reviewed-by: Leonardo Bras Suggested-by: Arnd Bergmann Link: https://lore.kernel.org/linux-riscv/CAK8P3a2rnz9mQqhN6-e0CGUUv9rntREL= Fdxt_weiD7FxH7fkfQ@mail.gmail.com/ Signed-off-by: Guo Ren Signed-off-by: Guo Ren Acked-by: Peter Zijlstra (Intel) Reviewed-by: Andrew Jones Reviewed-by: Andrea Parri --- include/asm-generic/spinlock.h | 14 +++++++------- include/asm-generic/spinlock_types.h | 12 ++---------- 2 files changed, 9 insertions(+), 17 deletions(-) diff --git a/include/asm-generic/spinlock.h b/include/asm-generic/spinlock.h index 90803a826ba0..4773334ee638 100644 --- a/include/asm-generic/spinlock.h +++ b/include/asm-generic/spinlock.h @@ -32,7 +32,7 @@ =20 static __always_inline void arch_spin_lock(arch_spinlock_t *lock) { - u32 val =3D atomic_fetch_add(1<<16, lock); + u32 val =3D atomic_fetch_add(1<<16, &lock->val); u16 ticket =3D val >> 16; =20 if (ticket =3D=3D (u16)val) @@ -46,31 +46,31 @@ static __always_inline void arch_spin_lock(arch_spinloc= k_t *lock) * have no outstanding writes due to the atomic_fetch_add() the extra * orderings are free. */ - atomic_cond_read_acquire(lock, ticket =3D=3D (u16)VAL); + atomic_cond_read_acquire(&lock->val, ticket =3D=3D (u16)VAL); smp_mb(); } =20 static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) { - u32 old =3D atomic_read(lock); + u32 old =3D atomic_read(&lock->val); =20 if ((old >> 16) !=3D (old & 0xffff)) return false; =20 - return atomic_try_cmpxchg(lock, &old, old + (1<<16)); /* SC, for RCsc */ + return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RC= sc */ } =20 static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) { u16 *ptr =3D (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); - u32 val =3D atomic_read(lock); + u32 val =3D atomic_read(&lock->val); =20 smp_store_release(ptr, (u16)val + 1); } =20 static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) { - u32 val =3D lock.counter; + u32 val =3D lock.val.counter; =20 return ((val >> 16) =3D=3D (val & 0xffff)); } @@ -84,7 +84,7 @@ static __always_inline int arch_spin_is_locked(arch_spinl= ock_t *lock) =20 static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) { - u32 val =3D atomic_read(lock); + u32 val =3D atomic_read(&lock->val); =20 return (s16)((val >> 16) - (val & 0xffff)) > 1; } diff --git a/include/asm-generic/spinlock_types.h b/include/asm-generic/spi= nlock_types.h index 8962bb730945..f534aa5de394 100644 --- a/include/asm-generic/spinlock_types.h +++ b/include/asm-generic/spinlock_types.h @@ -3,15 +3,7 @@ #ifndef __ASM_GENERIC_SPINLOCK_TYPES_H #define __ASM_GENERIC_SPINLOCK_TYPES_H =20 -#include -typedef atomic_t arch_spinlock_t; - -/* - * qrwlock_types depends on arch_spinlock_t, so we must typedef that befor= e the - * include. - */ -#include - -#define __ARCH_SPIN_LOCK_UNLOCKED ATOMIC_INIT(0) +#include +#include =20 #endif /* __ASM_GENERIC_SPINLOCK_TYPES_H */ --=20 2.39.2 From nobody Sun Nov 24 17:42:18 2024 Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E6825184549 for ; Sun, 3 Nov 2024 15:02:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730646133; cv=none; b=eMUnFJz5e2nd2gBgy2/RUaBjuEIi2pvo0ddHR4MWvpUFptN6pyN27Uw5pZucZnUW92N9RZ1zSnSQ6NDg/D3vJ3dHXQ5hUeH7nPcIhRZVE9HZ4DVWfvG5qVHzm4/dxuD1WXESg7JRBo2MwDd0x50sDKRVAih/YJzva00Nzy6CKeg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730646133; c=relaxed/simple; bh=dK1UxydJIu5czPL0vZxns1LgMAc0GDD61sIdYA37OY0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=RgSwG9VPKLZg0m7K56VaykAYaSq9dOWGMAVoQ5UCD64Rk5LaQVWf8lNt4xTqKTAZTzjWNEzphy4LgvYT2mkJZ8/lsYo39GSNPwO91YcrxYJUZ66yNXbKGZiOAG4baL25faE4BKOJQG+tzRUCTCQV5H6uBeGU1zCE17/DbG1OiLo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=q/M24nY0; arc=none smtp.client-ip=209.85.221.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="q/M24nY0" Received: by mail-wr1-f44.google.com with SMTP id ffacd0b85a97d-37d4c1b1455so2119004f8f.3 for ; Sun, 03 Nov 2024 07:02:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730646129; x=1731250929; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hiozRe0ZQqJk1GBz+SIEaweXO3Ezj4/Y82ena4X838E=; b=q/M24nY0QTbHMj2jQqw5FYcAzRNFBluTgQUF4bZ2Cpn/7dd8KUXO16cm0S/LPrmGGN HBYyYBGF4Yiw35VksAeVD8UuU/REW2QMZyEDI+g9GS1LbTFDgqGR5v8freeAWqtB80FJ 4xz4qVyizW064JvTCRzW1eG6fFzd77Svm7PpUbz3bZecaP9F7oG4QrzlPviy+TBR4BY6 TzaljB1MGMcCSOeuE19Je/6HMVFknGmnkppW/8AkGomI0FQAzO0SB/NMBaS3tO3Z0iMI sHlBTAFBzCTRpO4WgjcFNY9JI1jrO1N4TUIaP+09cYdSb/5aZRPOKyhZPlYJ/M11c5HK 9MVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730646129; x=1731250929; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hiozRe0ZQqJk1GBz+SIEaweXO3Ezj4/Y82ena4X838E=; b=HEThHCVlEWiu2+p8PKLATA8DiYoKfjKwmQqgeZhn38x10C1QOGN5cNEos1GFOqhRJM +iubVQVMS31aBHm2ewFuKvSIPHYd0xqlSNWHQWf3r9mWuLUrjal+nhnPykueZaFD5UT3 27aHKEBOdIXHgsKJEuNQ42pwZxP1yVCnzMeG1oThPKsq6/NcdTU1ZMWvXsUx09mKPYzi Zd3X/1CTsrNG0gEOOS19issf3ZsFzu2kff7bHzhOZsnUZIbVW8etRoyPqAGli3WdWv4P my26YaHz13zoIxkAa4fziI7Torz9XkYm4RN8SfObZTCTazmbX5gYw63LUA369owj/1// nQdw== X-Forwarded-Encrypted: i=1; AJvYcCU494ge7XgM91BYcHttCMSvEhEVCvdaW+DdVZbp4FXFJ94NdkeQ4p+463jxdiAqIizzF2EEWaZYAeMmxUM=@vger.kernel.org X-Gm-Message-State: AOJu0YzrJk5s4QEn1luSKAh07gbhix6Dp+nU3k80XbdU/KImOuL5Q4kG rWMrNAGhbZKtZd3+bnYE4KNIe3MBX0A14S0tNs+H1bYHys4qpDd0jpFgoyfrl+A= X-Google-Smtp-Source: AGHT+IEWaiYi7foaZ2KTG2kLPtUDJP2UndtLmP9dR8kSWRyDgH5SEK/ylfyL6hxCJksB0pHYX//hgw== X-Received: by 2002:a05:6000:1b08:b0:378:89be:1825 with SMTP id ffacd0b85a97d-381b710fbb0mr11405436f8f.49.1730646129165; Sun, 03 Nov 2024 07:02:09 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-381c10d4b82sm10818939f8f.43.2024.11.03.07.02.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 07:02:08 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Guo Ren , Andrew Jones Subject: [PATCH v6 10/13] asm-generic: ticket-lock: Add separate ticket-lock.h Date: Sun, 3 Nov 2024 15:51:50 +0100 Message-Id: <20241103145153.105097-11-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Guo Ren Add a separate ticket-lock.h to include multiple spinlock versions and select one at compile time or runtime. Reviewed-by: Leonardo Bras Suggested-by: Arnd Bergmann Link: https://lore.kernel.org/linux-riscv/CAK8P3a2rnz9mQqhN6-e0CGUUv9rntREL= Fdxt_weiD7FxH7fkfQ@mail.gmail.com/ Signed-off-by: Guo Ren Signed-off-by: Guo Ren Acked-by: Peter Zijlstra (Intel) Reviewed-by: Andrew Jones Reviewed-by: Andrea Parri --- include/asm-generic/spinlock.h | 87 +--------------------- include/asm-generic/ticket_spinlock.h | 103 ++++++++++++++++++++++++++ 2 files changed, 104 insertions(+), 86 deletions(-) create mode 100644 include/asm-generic/ticket_spinlock.h diff --git a/include/asm-generic/spinlock.h b/include/asm-generic/spinlock.h index 4773334ee638..970590baf61b 100644 --- a/include/asm-generic/spinlock.h +++ b/include/asm-generic/spinlock.h @@ -1,94 +1,9 @@ /* SPDX-License-Identifier: GPL-2.0 */ =20 -/* - * 'Generic' ticket-lock implementation. - * - * It relies on atomic_fetch_add() having well defined forward progress - * guarantees under contention. If your architecture cannot provide this, = stick - * to a test-and-set lock. - * - * It also relies on atomic_fetch_add() being safe vs smp_store_release() = on a - * sub-word of the value. This is generally true for anything LL/SC althou= gh - * you'd be hard pressed to find anything useful in architecture specifica= tions - * about this. If your architecture cannot do this you might be better off= with - * a test-and-set. - * - * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and = hence - * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along= with - * a full fence after the spin to upgrade the otherwise-RCpc - * atomic_cond_read_acquire(). - * - * The implementation uses smp_cond_load_acquire() to spin, so if the - * architecture has WFE like instructions to sleep instead of poll for word - * modifications be sure to implement that (see ARM64 for example). - * - */ - #ifndef __ASM_GENERIC_SPINLOCK_H #define __ASM_GENERIC_SPINLOCK_H =20 -#include -#include - -static __always_inline void arch_spin_lock(arch_spinlock_t *lock) -{ - u32 val =3D atomic_fetch_add(1<<16, &lock->val); - u16 ticket =3D val >> 16; - - if (ticket =3D=3D (u16)val) - return; - - /* - * atomic_cond_read_acquire() is RCpc, but rather than defining a - * custom cond_read_rcsc() here we just emit a full fence. We only - * need the prior reads before subsequent writes ordering from - * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we - * have no outstanding writes due to the atomic_fetch_add() the extra - * orderings are free. - */ - atomic_cond_read_acquire(&lock->val, ticket =3D=3D (u16)VAL); - smp_mb(); -} - -static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) -{ - u32 old =3D atomic_read(&lock->val); - - if ((old >> 16) !=3D (old & 0xffff)) - return false; - - return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RC= sc */ -} - -static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) -{ - u16 *ptr =3D (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); - u32 val =3D atomic_read(&lock->val); - - smp_store_release(ptr, (u16)val + 1); -} - -static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) -{ - u32 val =3D lock.val.counter; - - return ((val >> 16) =3D=3D (val & 0xffff)); -} - -static __always_inline int arch_spin_is_locked(arch_spinlock_t *lock) -{ - arch_spinlock_t val =3D READ_ONCE(*lock); - - return !arch_spin_value_unlocked(val); -} - -static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) -{ - u32 val =3D atomic_read(&lock->val); - - return (s16)((val >> 16) - (val & 0xffff)) > 1; -} - +#include #include =20 #endif /* __ASM_GENERIC_SPINLOCK_H */ diff --git a/include/asm-generic/ticket_spinlock.h b/include/asm-generic/ti= cket_spinlock.h new file mode 100644 index 000000000000..cfcff22b37b3 --- /dev/null +++ b/include/asm-generic/ticket_spinlock.h @@ -0,0 +1,103 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * 'Generic' ticket-lock implementation. + * + * It relies on atomic_fetch_add() having well defined forward progress + * guarantees under contention. If your architecture cannot provide this, = stick + * to a test-and-set lock. + * + * It also relies on atomic_fetch_add() being safe vs smp_store_release() = on a + * sub-word of the value. This is generally true for anything LL/SC althou= gh + * you'd be hard pressed to find anything useful in architecture specifica= tions + * about this. If your architecture cannot do this you might be better off= with + * a test-and-set. + * + * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and = hence + * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along= with + * a full fence after the spin to upgrade the otherwise-RCpc + * atomic_cond_read_acquire(). + * + * The implementation uses smp_cond_load_acquire() to spin, so if the + * architecture has WFE like instructions to sleep instead of poll for word + * modifications be sure to implement that (see ARM64 for example). + * + */ + +#ifndef __ASM_GENERIC_TICKET_SPINLOCK_H +#define __ASM_GENERIC_TICKET_SPINLOCK_H + +#include +#include + +static __always_inline void ticket_spin_lock(arch_spinlock_t *lock) +{ + u32 val =3D atomic_fetch_add(1<<16, &lock->val); + u16 ticket =3D val >> 16; + + if (ticket =3D=3D (u16)val) + return; + + /* + * atomic_cond_read_acquire() is RCpc, but rather than defining a + * custom cond_read_rcsc() here we just emit a full fence. We only + * need the prior reads before subsequent writes ordering from + * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we + * have no outstanding writes due to the atomic_fetch_add() the extra + * orderings are free. + */ + atomic_cond_read_acquire(&lock->val, ticket =3D=3D (u16)VAL); + smp_mb(); +} + +static __always_inline bool ticket_spin_trylock(arch_spinlock_t *lock) +{ + u32 old =3D atomic_read(&lock->val); + + if ((old >> 16) !=3D (old & 0xffff)) + return false; + + return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RC= sc */ +} + +static __always_inline void ticket_spin_unlock(arch_spinlock_t *lock) +{ + u16 *ptr =3D (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); + u32 val =3D atomic_read(&lock->val); + + smp_store_release(ptr, (u16)val + 1); +} + +static __always_inline int ticket_spin_value_unlocked(arch_spinlock_t lock) +{ + u32 val =3D lock.val.counter; + + return ((val >> 16) =3D=3D (val & 0xffff)); +} + +static __always_inline int ticket_spin_is_locked(arch_spinlock_t *lock) +{ + arch_spinlock_t val =3D READ_ONCE(*lock); + + return !ticket_spin_value_unlocked(val); +} + +static __always_inline int ticket_spin_is_contended(arch_spinlock_t *lock) +{ + u32 val =3D atomic_read(&lock->val); + + return (s16)((val >> 16) - (val & 0xffff)) > 1; +} + +/* + * Remapping spinlock architecture specific functions to the corresponding + * ticket spinlock functions. + */ +#define arch_spin_is_locked(l) ticket_spin_is_locked(l) +#define arch_spin_is_contended(l) ticket_spin_is_contended(l) +#define arch_spin_value_unlocked(l) ticket_spin_value_unlocked(l) +#define arch_spin_lock(l) ticket_spin_lock(l) +#define arch_spin_trylock(l) ticket_spin_trylock(l) +#define arch_spin_unlock(l) ticket_spin_unlock(l) + +#endif /* __ASM_GENERIC_TICKET_SPINLOCK_H */ --=20 2.39.2 From nobody Sun Nov 24 17:42:18 2024 Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 337B517C9E8 for ; Sun, 3 Nov 2024 15:03:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730646193; cv=none; b=NlJ+JvR8s/Ol30DBsPmKIs6UIKiCmfFCLhgXQ0ciGU56DyN29HG56Q4Zss4fZfedyPQZLN8cCi2Kparw2kbrnR8+uUz2GxkiFaFsjuHYEOAJSISyWRDZE8VOaxxUnQzutbnjWfULbgNbC++gb7lFUSXVedXPR8Ba3imFl5qHW0U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730646193; c=relaxed/simple; bh=xeHnByhcs5nCaAI8R1ijkMCW2JIwlS4zwscmM/llep4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=uko/wT2mDVgF4IJQHLPxsiphvIBeEwMCeClvDSbBEY69QZ9L6qZa4eT9i0mz+z1QN3JO+G0Zdz7pVrL82si103Omhkli8tz1Rl3Me45vEoeFCQtzsRaAVdvMD265PFIH+aBNpNpVZVH/bbsUrABXM1jGKysPxZYr6UnOSeutr/I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=fLXQNess; arc=none smtp.client-ip=209.85.128.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="fLXQNess" Received: by mail-wm1-f46.google.com with SMTP id 5b1f17b1804b1-43167ff0f91so29820295e9.1 for ; Sun, 03 Nov 2024 07:03:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730646190; x=1731250990; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=51tYDRwP1WvxpzmCthvL0bau6hmUz8qjqFRqrLI9cd4=; b=fLXQNessp7bw7zyHRTo7WBFUD2v78z//Wh0lh865XQudosktnxWmbuqumUH7GLtYi/ PSuPAfrsw8srPjG7MK2V1xTLyT7tMhVnpIPK1ez2rDprDbcYycAPics22yh37pbhIS6F 2PF1XsLVtsChtyq15/QI3SNPmubWAOK7OvR7awNc/qJmdJRP5471CrvTMQw0HuYD8BhM XHakXShR0aW5sUpSls7hUHB0aYu1hbPEmHqu0CB+OzAPTpRZXsGhB+GLh1sk+vshjBB8 AksswBCQ0mnss+VowigWLTIuJKNcIvJnGflg/uoJhIukkMBNdThlEhhDRrWobHD/YWGE U5aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730646190; x=1731250990; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=51tYDRwP1WvxpzmCthvL0bau6hmUz8qjqFRqrLI9cd4=; b=HklNSwyomKe3kXPmuulmhEp4pBUsulFfSJWjMNyHoiFkaOFZnR11cgE0O2KP2FKu9G q+04xTrSqeO/jDyMn+MJKVYSSnusPZl+beGo/0nWJVpkOUxfnMRhtaumNvXVnixT/qos zt34Kj86+lxbz1BqqAVOVOULnrMNPiyqj3Ldq4ykX/k2VK1NgrCfiOFIQEr8QcTSqBIP DNTl9zc9LeJaytdztkgGn4PxIqB8vnKRAy1xH/URTYD40EM/ev2GHOT00Wf+ksr63gZn h3sgbrbEt0jjRjAO8KON1S+nFRyElIC/ys63ndMeJyVQRaoV7cwqKaQPNdTA7tuDGuwa J5eg== X-Forwarded-Encrypted: i=1; AJvYcCU5Q8HRPRSnDH6r/gUbipSry0RtYkxBHgeFWI3M/bjaJnVfPbnCmF7s4KCQNFQvX3+9wDSQo0MxUXJul5g=@vger.kernel.org X-Gm-Message-State: AOJu0YzYxKHFBTi2O9AHkcEhz2dcWELB4R2oINAl4Y2JPzNkeGPLtjfI 5zAYsaxLz+skaqQJu1Ln7qfV4MKCAz1XBkEuSAUk/swI0TZq/ekJOgKm0++J6xA= X-Google-Smtp-Source: AGHT+IFqldFYKHqBiRaB6ae6bURZkYwVuzbn5bQ+e+Hym9E5heEep9SJOfB/eoXxuGPgZenIG/xzCA== X-Received: by 2002:a05:600c:1c9e:b0:431:5632:448b with SMTP id 5b1f17b1804b1-4327b80d3c5mr115258455e9.25.1730646190486; Sun, 03 Nov 2024 07:03:10 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-381c113e5cbsm10816042f8f.80.2024.11.03.07.03.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 07:03:10 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v6 11/13] riscv: Add ISA extension parsing for Ziccrse Date: Sun, 3 Nov 2024 15:51:51 +0100 Message-Id: <20241103145153.105097-12-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add support to parse the Ziccrse string in the riscv,isa string. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones Reviewed-by: Andrea Parri --- arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/kernel/cpufeature.c | 1 + 2 files changed, 2 insertions(+) diff --git a/arch/riscv/include/asm/hwcap.h b/arch/riscv/include/asm/hwcap.h index 74bcb0e2bd1f..0aa3c3f5e682 100644 --- a/arch/riscv/include/asm/hwcap.h +++ b/arch/riscv/include/asm/hwcap.h @@ -94,6 +94,7 @@ #define RISCV_ISA_EXT_ZAWRS 85 #define RISCV_ISA_EXT_SVVPTC 86 #define RISCV_ISA_EXT_ZABHA 87 +#define RISCV_ISA_EXT_ZICCRSE 88 =20 #define RISCV_ISA_EXT_XLINUXENVCFG 127 =20 diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index 5e743d8d34f5..5f453a039ec9 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -314,6 +314,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] =3D { riscv_ext_zicbom_validate), __RISCV_ISA_EXT_SUPERSET_VALIDATE(zicboz, RISCV_ISA_EXT_ZICBOZ, riscv_xli= nuxenvcfg_exts, riscv_ext_zicboz_validate), + __RISCV_ISA_EXT_DATA(ziccrse, RISCV_ISA_EXT_ZICCRSE), __RISCV_ISA_EXT_DATA(zicntr, RISCV_ISA_EXT_ZICNTR), __RISCV_ISA_EXT_DATA(zicond, RISCV_ISA_EXT_ZICOND), __RISCV_ISA_EXT_DATA(zicsr, RISCV_ISA_EXT_ZICSR), --=20 2.39.2 From nobody Sun Nov 24 17:42:18 2024 Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6A2B417DE15 for ; Sun, 3 Nov 2024 15:04:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730646255; cv=none; b=loDbvZKVBXW3/2MPiTHw79E51rd64vu+Ya30jWk40HRzomZcqGpLuMLveTpiyFkEwIXegPZx9FSkHwCB9SWnnR00Ds3DqUOmKQpUkHBedZwrm9KFsTvKU9Pbwwmk+83UlJb6LfKIW8Z18EyYDq8PURZaXRelIjOtbnh9GFuHP2A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730646255; c=relaxed/simple; bh=DDI2zIjw9AIG8rhpAgDtY+aKq0SOPiidb9JfsFGBdSY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=UblJkY2Yk7oaOq3IIEoizxr08RoGGa4L/OPm/gZZ4bfI06+IihPWULsBZI2HOfS+rUycSX10KpHiAf4qIj/HxViudYQufn1BUkl3o8wg/0r3WeQJ/D3R/fXVoLveRZc2ldQjkUxu9Av9J1A1iBQiedgmUkbZ+nCwZaFbaD4QIg4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=vEo2u1/S; arc=none smtp.client-ip=209.85.128.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="vEo2u1/S" Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-43193678216so29773315e9.0 for ; Sun, 03 Nov 2024 07:04:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730646252; x=1731251052; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CYOR6ehP8ZUAtZy2SlX+P5w8jiHdus9J4uApBvZwtJU=; b=vEo2u1/SsXzwb6c1sxGlxeEGJO3XSMf6TUC4IpOAU6NTove6jX6XfgQ0DK3L7quoKc fVO4LdDRytFgoLxE2GgfJbUzaBmV4HLpsw0ZlstCZtxyA+xbvU80+nDqOg8HCnjEvjDC ujqbaROapfIS1DqDVvd9ZzRura7stsd86lVDomuRO6yMK3dXtAB/dNN8IzBW+u/bM95I ifTTEfRSjMCGxTC3h90sfXYeUJoDOaBDtGL48gMB4pdtjuQI29yc/OFE6k2PaaBXrOIQ kz4/bMwNFcP0oF+XF9BoiJKmRpO6C8Wuov9SEsjwgJ4PZHxWF2OjfFqX9Q6MZUvyJKb1 GxfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730646252; x=1731251052; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CYOR6ehP8ZUAtZy2SlX+P5w8jiHdus9J4uApBvZwtJU=; b=CBx50TpnjKEa/WXsFqZSOKN6auU1nFX+sx9r+mJITv9mCn6iOB2hueRWW5eTktcH57 dDQNnMbo86R19G5/DsPaNbVKcQfAOHoZJsnrGnhFIimWzWLEU+BjxOidkI0dcvGvR/Lg duQUymaAeOuOOwfYKbP6LT21h6GA4U8OUcCMBw8JjcnJ3v1yHePKXe2P9VrucEeODQoQ s1PD85CEJIXFcKjnMW2jpZOFBPlr7DhLTgFZzKJBsUa16bLjhfxBPIgYfI/+VYn75+Tb VFwlGQpiLRPvVRS2Qa7c8+jZa2LHtzUL/kl1jwi+bF+wE3VP+y2GPEjbqVZIa6h0nxpm L8AA== X-Forwarded-Encrypted: i=1; AJvYcCWAAqBckrC0Qr0dJt+Y23QK70kFwNoNuNH+XOqBWICp3/CX+rFhKkhT7l+zoO0eQC2NZvlCbTZbp7lQFzc=@vger.kernel.org X-Gm-Message-State: AOJu0YxM6FufiYMLJhjhV4UmPjsvQ+jaQgl/fqxjY3W7TV6Y65IGZeei ET0+3tmwjsMzYlo+LenIzG+6B4nh/pgxKCFYezxrnazeU2C/97oH9lTdLEQnpv4= X-Google-Smtp-Source: AGHT+IHLb+bX4JOWUSFmTs6UYPJV8WTvK/tBJRvIgcA7ae04ZGiofBjbX29h/oO76eksCbmobaHszg== X-Received: by 2002:a05:600c:4fcb:b0:426:5269:1a50 with SMTP id 5b1f17b1804b1-4319ac9c15fmr240004605e9.11.1730646251731; Sun, 03 Nov 2024 07:04:11 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-381c10b7b32sm10858711f8f.18.2024.11.03.07.04.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 07:04:11 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Conor Dooley , Andrew Jones Subject: [PATCH v6 12/13] dt-bindings: riscv: Add Ziccrse ISA extension description Date: Sun, 3 Nov 2024 15:51:52 +0100 Message-Id: <20241103145153.105097-13-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add description for the Ziccrse ISA extension which was ratified in the riscv profiles specification v1.0. Signed-off-by: Alexandre Ghiti Reviewed-by: Guo Ren Acked-by: Conor Dooley Reviewed-by: Andrew Jones Reviewed-by: Andrea Parri --- Documentation/devicetree/bindings/riscv/extensions.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Documentation/devicetree/bindings/riscv/extensions.yaml b/Docu= mentation/devicetree/bindings/riscv/extensions.yaml index db062107823b..35a9ad1d7e63 100644 --- a/Documentation/devicetree/bindings/riscv/extensions.yaml +++ b/Documentation/devicetree/bindings/riscv/extensions.yaml @@ -296,6 +296,12 @@ properties: in commit 64074bc ("Update version numbers for Zfh/Zfinx") of riscv-isa-manual. =20 + - const: ziccrse + description: + The standard Ziccrse extension which provides forward progress + guarantee on LR/SC sequences, as ratified in commit b1d806605f= 87 + ("Updated to ratified state.") of the riscv profiles specifica= tion. + - const: zk description: The standard Zk Standard Scalar cryptography extension as rati= fied --=20 2.39.2 From nobody Sun Nov 24 17:42:18 2024 Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DC66B175D56 for ; Sun, 3 Nov 2024 15:05:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730646316; cv=none; b=Si68ChR+/yJH3cR+4ooffz+zb6Nn7I9osWyIiQfBaX5Ckc0SeM2r+OK7lJ+mRb4P16YpPm4j5FgpKwWmQRUUK/xlUQeyRZ734Y5u4Q48nbAvYPo1D9xZmFfsGUqCNoZihcwZ3ZzEUIrasFYpC3I89TuFr2Gm7+Z5bDT1IaX6P5Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730646316; c=relaxed/simple; bh=5uZKIfy4UVr5gXxfmoEq12WZA8O8TUbAyVEPjJGGnOU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=mfxvjQ38JL1JcV3R6k5MvNttobu+PCvNs6WdwYJO3oEMrhlo8slHRR3G4GVoAW5v9a71fPplsWPA/B1yAaLTQWvN/dzhLnYc8sxoL9gVbbOhJgOPddj5HYpg1TEGEEcI1X4AJKvWhek9Dl33J7GQLjQ3LbHRzGc9v9ADg4Jw2tQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=LSvZRAni; arc=none smtp.client-ip=209.85.128.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="LSvZRAni" Received: by mail-wm1-f46.google.com with SMTP id 5b1f17b1804b1-431ac30d379so27662075e9.1 for ; Sun, 03 Nov 2024 07:05:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730646313; x=1731251113; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9aY9phw0h9f3xNjCboMfKLftvRP8BQ6/29lVmcfPzaI=; b=LSvZRAnimy5AsgqwT1+zLdQl/yj4bkLr361Lvj6oSCqG8JRjADXgj2IurXaCjjpl/I vtodkR5q9/ZyyMWTbepicr4x3WJ7vognpL7LCr8NAmAMfDDhXNdvU+cJk99MeG8/ngkf N6QDo2ohlMgnlEb/O3j8Ms15KcqFtU/sPCxsUElhWlmvDue9Xhf7Qmn21akV0spzcKMw aMJf7ZG5huvqo22ZdSLZkMZNU8Kb3kOgrUO5f52N1p/DhjbdDnsM1VwP7QxiwBCozsYB YntZ3KABpwki4yxV2q3MeT9bwfadFPc7s/pGp6ZVHPXWhv+IDKLLbEVTLMl/rOjEY7U7 jcFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730646313; x=1731251113; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9aY9phw0h9f3xNjCboMfKLftvRP8BQ6/29lVmcfPzaI=; b=YKEOcKmzbwG4C5Xh8znhaNHnIMJ1cZZjLBQkOiDDktBipTjxfn1TnxIvvGFor2zDY8 Ukr2J/mRnjqhPE4EfzYn162jmaRa32IQEic69sbfWt0ILjIWOYt5OTphEJejynIl3AXI LmCHZZrpgBrpuW1OwLbVoN7orYkFrTbNtizSw/sAMJ5FzK7yU8AEnyeQUVltaAgwJwSv IBqiJ7CKelfk+rBxeq0LVQiwR/FQdNCf1dpQ6Cs2TnQhiyW0/3d9AeOM6CKHiF22rHER su8lAgnvTj6Z33aevQOmNaK+F4Mtx+iHhjJo1obNuFtHjw8YOV5WXS/XZDpRBaEAp1O7 OnXw== X-Forwarded-Encrypted: i=1; AJvYcCWY703nwtyBquMkB7qn6MfrmspUT/Jp7SRWKXnR+dkV2tm58TSXtSB/R9CkrtMSQC0wNWWQhM70MXS0rMQ=@vger.kernel.org X-Gm-Message-State: AOJu0YxqlNRm0c3Jbl1WdtoXeLrL4Yp6PuKqGQweQC/EcL23ZhybExs7 F5AGy4rsEK4TrUU8KNntMuvGhRxlFZuOyFfQ1rT+6ZeQESpMEvbu15/uBKfx18I= X-Google-Smtp-Source: AGHT+IHPZiSZzcaPz06TDTdXTUesvEzSWH1SxGmB9LrKBfkdnmynnmf9paGsFdY/fWwSFbEmyLWwew== X-Received: by 2002:a05:600c:45cd:b0:431:5eeb:2214 with SMTP id 5b1f17b1804b1-4319ad363eamr237927625e9.33.1730646313170; Sun, 03 Nov 2024 07:05:13 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-381c113e89csm10899063f8f.74.2024.11.03.07.05.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 07:05:12 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v6 13/13] riscv: Add qspinlock support Date: Sun, 3 Nov 2024 15:51:53 +0100 Message-Id: <20241103145153.105097-14-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In order to produce a generic kernel, a user can select CONFIG_COMBO_SPINLOCKS which will fallback at runtime to the ticket spinlock implementation if Zabha or Ziccrse are not present. Note that we can't use alternatives here because the discovery of extensions is done too late and we need to start with the qspinlock implementation because the ticket spinlock implementation would pollute the spinlock value, so let's use static keys. This is largely based on Guo's work and Leonardo reviews at [1]. Link: https://lore.kernel.org/linux-riscv/20231225125847.2778638-1-guoren@k= ernel.org/ [1] Signed-off-by: Guo Ren Signed-off-by: Alexandre Ghiti Reviewed-by: Andrea Parri --- .../locking/queued-spinlocks/arch-support.txt | 2 +- arch/riscv/Kconfig | 34 ++++++++++++++ arch/riscv/include/asm/Kbuild | 4 +- arch/riscv/include/asm/spinlock.h | 47 +++++++++++++++++++ arch/riscv/kernel/setup.c | 37 +++++++++++++++ include/asm-generic/qspinlock.h | 2 + include/asm-generic/ticket_spinlock.h | 2 + 7 files changed, 126 insertions(+), 2 deletions(-) create mode 100644 arch/riscv/include/asm/spinlock.h diff --git a/Documentation/features/locking/queued-spinlocks/arch-support.t= xt b/Documentation/features/locking/queued-spinlocks/arch-support.txt index 22f2990392ff..cf26042480e2 100644 --- a/Documentation/features/locking/queued-spinlocks/arch-support.txt +++ b/Documentation/features/locking/queued-spinlocks/arch-support.txt @@ -20,7 +20,7 @@ | openrisc: | ok | | parisc: | TODO | | powerpc: | ok | - | riscv: | TODO | + | riscv: | ok | | s390: | TODO | | sh: | TODO | | sparc: | ok | diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 093ee6537331..f5698ecc5ccc 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -82,6 +82,7 @@ config RISCV select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP select ARCH_WANTS_NO_INSTR select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE + select ARCH_WEAK_RELEASE_ACQUIRE if ARCH_USE_QUEUED_SPINLOCKS select BINFMT_FLAT_NO_DATA_START_OFFSET if !MMU select BUILDTIME_TABLE_SORT if MMU select CLINT_TIMER if RISCV_M_MODE @@ -507,6 +508,39 @@ config NODES_SHIFT Specify the maximum number of NUMA Nodes available on the target system. Increases memory reserved to accommodate various tables. =20 +choice + prompt "RISC-V spinlock type" + default RISCV_COMBO_SPINLOCKS + +config RISCV_TICKET_SPINLOCKS + bool "Using ticket spinlock" + +config RISCV_QUEUED_SPINLOCKS + bool "Using queued spinlock" + depends on SMP && MMU && NONPORTABLE + select ARCH_USE_QUEUED_SPINLOCKS + help + The queued spinlock implementation requires the forward progress + guarantee of cmpxchg()/xchg() atomic operations: CAS with Zabha or + LR/SC with Ziccrse provide such guarantee. + + Select this if and only if Zabha or Ziccrse is available on your + platform, RISCV_QUEUED_SPINLOCKS must not be selected for platforms + without one of those extensions. + + If unsure, select RISCV_COMBO_SPINLOCKS, which will use qspinlocks + when supported and otherwise ticket spinlocks. + +config RISCV_COMBO_SPINLOCKS + bool "Using combo spinlock" + depends on SMP && MMU + select ARCH_USE_QUEUED_SPINLOCKS + help + Embed both queued spinlock and ticket lock so that the spinlock + implementation can be chosen at runtime. + +endchoice + config RISCV_ALTERNATIVE bool depends on !XIP_KERNEL diff --git a/arch/riscv/include/asm/Kbuild b/arch/riscv/include/asm/Kbuild index 1461af12da6e..de13d5a234f8 100644 --- a/arch/riscv/include/asm/Kbuild +++ b/arch/riscv/include/asm/Kbuild @@ -6,10 +6,12 @@ generic-y +=3D early_ioremap.h generic-y +=3D flat.h generic-y +=3D kvm_para.h generic-y +=3D mmzone.h +generic-y +=3D mcs_spinlock.h generic-y +=3D parport.h -generic-y +=3D spinlock.h generic-y +=3D spinlock_types.h +generic-y +=3D ticket_spinlock.h generic-y +=3D qrwlock.h generic-y +=3D qrwlock_types.h +generic-y +=3D qspinlock.h generic-y +=3D user.h generic-y +=3D vmlinux.lds.h diff --git a/arch/riscv/include/asm/spinlock.h b/arch/riscv/include/asm/spi= nlock.h new file mode 100644 index 000000000000..e5121b89acea --- /dev/null +++ b/arch/riscv/include/asm/spinlock.h @@ -0,0 +1,47 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __ASM_RISCV_SPINLOCK_H +#define __ASM_RISCV_SPINLOCK_H + +#ifdef CONFIG_RISCV_COMBO_SPINLOCKS +#define _Q_PENDING_LOOPS (1 << 9) + +#define __no_arch_spinlock_redefine +#include +#include +#include + +/* + * TODO: Use an alternative instead of a static key when we are able to pa= rse + * the extensions string earlier in the boot process. + */ +DECLARE_STATIC_KEY_TRUE(qspinlock_key); + +#define SPINLOCK_BASE_DECLARE(op, type, type_lock) \ +static __always_inline type arch_spin_##op(type_lock lock) \ +{ \ + if (static_branch_unlikely(&qspinlock_key)) \ + return queued_spin_##op(lock); \ + return ticket_spin_##op(lock); \ +} + +SPINLOCK_BASE_DECLARE(lock, void, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(unlock, void, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(is_locked, int, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(is_contended, int, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(trylock, bool, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(value_unlocked, int, arch_spinlock_t) + +#elif defined(CONFIG_RISCV_QUEUED_SPINLOCKS) + +#include + +#else + +#include + +#endif + +#include + +#endif /* __ASM_RISCV_SPINLOCK_H */ diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c index a2cde65b69e9..438e4f6ad2ad 100644 --- a/arch/riscv/kernel/setup.c +++ b/arch/riscv/kernel/setup.c @@ -244,6 +244,42 @@ static void __init parse_dtb(void) #endif } =20 +#if defined(CONFIG_RISCV_COMBO_SPINLOCKS) +DEFINE_STATIC_KEY_TRUE(qspinlock_key); +EXPORT_SYMBOL(qspinlock_key); +#endif + +static void __init riscv_spinlock_init(void) +{ + char *using_ext =3D NULL; + + if (IS_ENABLED(CONFIG_RISCV_TICKET_SPINLOCKS)) { + pr_info("Ticket spinlock: enabled\n"); + return; + } + + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && + IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && + riscv_isa_extension_available(NULL, ZABHA) && + riscv_isa_extension_available(NULL, ZACAS)) { + using_ext =3D "using Zabha"; + } else if (riscv_isa_extension_available(NULL, ZICCRSE)) { + using_ext =3D "using Ziccrse"; + } +#if defined(CONFIG_RISCV_COMBO_SPINLOCKS) + else { + static_branch_disable(&qspinlock_key); + pr_info("Ticket spinlock: enabled\n"); + return; + } +#endif + + if (!using_ext) + pr_err("Queued spinlock without Zabha or Ziccrse"); + else + pr_info("Queued spinlock %s: enabled\n", using_ext); +} + extern void __init init_rt_signal_env(void); =20 void __init setup_arch(char **cmdline_p) @@ -297,6 +333,7 @@ void __init setup_arch(char **cmdline_p) riscv_set_dma_cache_alignment(); =20 riscv_user_isa_enable(); + riscv_spinlock_init(); } =20 bool arch_cpu_is_hotpluggable(int cpu) diff --git a/include/asm-generic/qspinlock.h b/include/asm-generic/qspinloc= k.h index 0655aa5b57b2..bf47cca2c375 100644 --- a/include/asm-generic/qspinlock.h +++ b/include/asm-generic/qspinlock.h @@ -136,6 +136,7 @@ static __always_inline bool virt_spin_lock(struct qspin= lock *lock) } #endif =20 +#ifndef __no_arch_spinlock_redefine /* * Remapping spinlock architecture specific functions to the corresponding * queued spinlock functions. @@ -146,5 +147,6 @@ static __always_inline bool virt_spin_lock(struct qspin= lock *lock) #define arch_spin_lock(l) queued_spin_lock(l) #define arch_spin_trylock(l) queued_spin_trylock(l) #define arch_spin_unlock(l) queued_spin_unlock(l) +#endif =20 #endif /* __ASM_GENERIC_QSPINLOCK_H */ diff --git a/include/asm-generic/ticket_spinlock.h b/include/asm-generic/ti= cket_spinlock.h index cfcff22b37b3..325779970d8a 100644 --- a/include/asm-generic/ticket_spinlock.h +++ b/include/asm-generic/ticket_spinlock.h @@ -89,6 +89,7 @@ static __always_inline int ticket_spin_is_contended(arch_= spinlock_t *lock) return (s16)((val >> 16) - (val & 0xffff)) > 1; } =20 +#ifndef __no_arch_spinlock_redefine /* * Remapping spinlock architecture specific functions to the corresponding * ticket spinlock functions. @@ -99,5 +100,6 @@ static __always_inline int ticket_spin_is_contended(arch= _spinlock_t *lock) #define arch_spin_lock(l) ticket_spin_lock(l) #define arch_spin_trylock(l) ticket_spin_trylock(l) #define arch_spin_unlock(l) ticket_spin_unlock(l) +#endif =20 #endif /* __ASM_GENERIC_TICKET_SPINLOCK_H */ --=20 2.39.2