From nobody Fri Dec 19 21:14:37 2025 Received: from mail-lj1-f171.google.com (mail-lj1-f171.google.com [209.85.208.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EC953171075 for ; Tue, 28 May 2024 15:11:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716909119; cv=none; b=VNXdzzdlm/kAsULhvTGJZfvrDhmQ18YntmNpiF3yvTZQGpOcWbdXxEZXF1g8GJhKIhm/dpA+7y4/b4gi2Ank6bDDoS9G+QwVNGjw8bN3izxtcKZcBbQXB2xnyOAKkhl+ncWkrUtXPA+C+Y5H39pyRQZJNlp7wx88K/Vw1RXWA74= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716909119; c=relaxed/simple; bh=TEvDOfsnylXJLXAYcAcZM/8r3TjyXkyenEf233xm5J0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Zldx/efqWyOF/WGKZ/JH2zWxyJgSC5Z6/zd5Fx9SPIpotCTMixDDa2ZmoZDGehQdW5px+yJhFfWLYGi7e7YljlSGWYI8uJVN+b7I4N9IrHiQ5RHmb0hFGAIJAoWFOnBHPS+VjDe8QtBRTxUYHKH3gs2/uGDFY1l71r9H4PXv+dA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=eJ7V4iUs; arc=none smtp.client-ip=209.85.208.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="eJ7V4iUs" Received: by mail-lj1-f171.google.com with SMTP id 38308e7fff4ca-2e72b8931caso10299391fa.0 for ; Tue, 28 May 2024 08:11:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1716909116; x=1717513916; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gqY2tac92Zgdj6EN5vhDUCiGPySLtAUcYkNNT11YFiQ=; b=eJ7V4iUs+u1NnkYTGo24gwCYXi5U46c5IbP0vk+xe5wjvK9WZhT4O8+h54B5n919yV KSt9PnI+QF5HfDTN9dStse+LcOII6iQd993KmtNBFaOSehF1EPz9LwPriXx+4lqJMO6X /JhjXCSTZCK0d3XSuOs/M7ctdA0TIcqH92pDrqCfFakgs7pSktkTttAp416/XrjtosPu 7U4Tk6r2udWMO0T5ddNP/aFYOMMc9l94F9QyLAhjv3PcKx5IsbZi4DoR4nRsMuKH6BsH 4qlLpcBRqmWbFS579fZnSEkQcBXEiSxA45f7lq8QbCSEEuH5WL2aH5M8YoJshKkDQWfV a22w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716909116; x=1717513916; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gqY2tac92Zgdj6EN5vhDUCiGPySLtAUcYkNNT11YFiQ=; b=UksMBRZnF7zsDX7eWLdQzBxMaUTiOddbTwmeX+/XPkwozWQqiLPq6cQFQbmUbuVxRV ecJP/pNaHkQ+fZ/QcdImyzqtFucgmIi33ma+kanX7Jg+IwCTRFDpLSoo7q7XF/uW7QVt m2hD/zn99h/M1oWxVfCjUMFUYoKmHxzwKgTXT/sTQHsrSZ0f0YbWJjyjotm5oRrisohs X1IQqZlU03m7V/jOETzzamI0dEruEWG9ZlThlwGWq0KyshB2zFC2p8gcUFZsISWWGLfy UGWjQzbDf/q9Jh0LsTlhPg3REyNZ455FGY4fCRMl1uXwr94aAdZ1lAmrjmzs7ybBqmV/ 9m/w== X-Forwarded-Encrypted: i=1; AJvYcCXTjDCvsm6INE3q4cLakKvYVohl7jAXVObNMJGyGJ5M7jnw58bTuYbPKm065SOjDsHQgQWguICiTd0EPKTkQH4WWcUtVmapYnAev4KR X-Gm-Message-State: AOJu0YyxC8nkVGafc6IB4hIvy9rDGc56/3+Rx8JctbPzTaQDJhQLGjZk FnyijcdWxLask9QtiFOfrbfw8XRaE/B8mTtBA5fOlf2eaNmoMpJyo1vycEDv97Q= X-Google-Smtp-Source: AGHT+IHCYhHKhCYiIOYwsJcUm7AY7hiBzh9j+j/t9IO5zqkvCwOqxf6clogJ5Pwo0VQvo1ymN8/SHQ== X-Received: by 2002:a2e:b0ed:0:b0:2e7:28ea:3c9d with SMTP id 38308e7fff4ca-2e95b2792f6mr65981801fa.51.1716909115927; Tue, 28 May 2024 08:11:55 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42100ef1207sm176823805e9.10.2024.05.28.08.11.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 May 2024 08:11:55 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH 1/7] riscv: Implement cmpxchg32/64() using Zacas Date: Tue, 28 May 2024 17:10:46 +0200 Message-Id: <20240528151052.313031-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240528151052.313031-1-alexghiti@rivosinc.com> References: <20240528151052.313031-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This adds runtime support for Zacas in cmpxchg operations. Signed-off-by: Alexandre Ghiti --- arch/riscv/Kconfig | 17 +++++++++++++++++ arch/riscv/Makefile | 11 +++++++++++ arch/riscv/include/asm/cmpxchg.h | 23 ++++++++++++++++++++--- 3 files changed, 48 insertions(+), 3 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 8a0f403432e8..b443def70139 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -579,6 +579,23 @@ config RISCV_ISA_V_PREEMPTIVE preemption. Enabling this config will result in higher memory consumption due to the allocation of per-task's kernel Vector context. =20 +config TOOLCHAIN_HAS_ZACAS + bool + default y + depends on !64BIT || $(cc-option,-mabi=3Dlp64 -march=3Drv64ima_zacas) + depends on !32BIT || $(cc-option,-mabi=3Dilp32 -march=3Drv32ima_zacas) + depends on AS_HAS_OPTION_ARCH + +config RISCV_ISA_ZACAS + bool "Zacas extension support for atomic CAS" + depends on TOOLCHAIN_HAS_ZACAS + default y + help + Adds support to use atomic CAS instead of LR/SC to implement kernel + atomic cmpxchg operation. + + If you don't know what to do here, say Y. + config TOOLCHAIN_HAS_ZBB bool default y diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index 5b3115a19852..d5b60b87998c 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -78,6 +78,17 @@ endif # Check if the toolchain supports Zihintpause extension riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZIHINTPAUSE) :=3D $(riscv-march-y)_zihi= ntpause =20 +# Check if the toolchain supports Zacas +ifdef CONFIG_AS_IS_LLVM +# Support for experimental Zacas was merged in LLVM 17, but the removal of +# the "experimental" was merged in LLVM 19. +KBUILD_CFLAGS +=3D -menable-experimental-extensions +KBUILD_AFLAGS +=3D -menable-experimental-extensions +riscv-march-y :=3D $(riscv-march-y)_zacas1p0 +else +riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZACAS) :=3D $(riscv-march-y)_zacas +endif + # Remove F,D,V from isa string for all. Keep extensions between "fd" and "= v" by # matching non-v and non-multi-letter extensions out with the filter ([^v_= ]*) KBUILD_CFLAGS +=3D -march=3D$(shell echo $(riscv-march-y) | sed -E 's/(rv3= 2ima|rv64ima)fd([^v_]*)v?/\1\2/') diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index 4d23f0c35b94..1c50b4821ac8 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -9,6 +9,7 @@ #include =20 #include +#include =20 #define __arch_xchg_masked(prepend, append, r, p, n) \ ({ \ @@ -132,21 +133,37 @@ r =3D (__typeof__(*(p)))((__retx & __mask) >> __s); \ }) =20 -#define __arch_cmpxchg(lr_sfx, sc_sfx, prepend, append, r, p, co, o, n) \ +#define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n= ) \ ({ \ + __label__ zacas, end; \ register unsigned int __rc; \ \ + asm goto(ALTERNATIVE("nop", "j %[zacas]", 0, \ + RISCV_ISA_EXT_ZACAS, 1) \ + : : : : zacas); \ + \ __asm__ __volatile__ ( \ prepend \ "0: lr" lr_sfx " %0, %2\n" \ " bne %0, %z3, 1f\n" \ - " sc" sc_sfx " %1, %z4, %2\n" \ + " sc" sc_cas_sfx " %1, %z4, %2\n" \ " bnez %1, 0b\n" \ append \ "1:\n" \ : "=3D&r" (r), "=3D&r" (__rc), "+A" (*(p)) \ : "rJ" (co o), "rJ" (n) \ : "memory"); \ + goto end; \ + \ +zacas: \ + __asm__ __volatile__ ( \ + prepend \ + " amocas" sc_cas_sfx " %0, %z2, %1\n" \ + append \ + : "+&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ +end: \ }) =20 #define _arch_cmpxchg(ptr, old, new, sc_sfx, prepend, append) \ @@ -154,7 +171,7 @@ __typeof__(ptr) __ptr =3D (ptr); \ __typeof__(*(__ptr)) __old =3D (old); \ __typeof__(*(__ptr)) __new =3D (new); \ - __typeof__(*(__ptr)) __ret; \ + __typeof__(*(__ptr)) __ret =3D (old); \ \ switch (sizeof(*__ptr)) { \ case 1: \ --=20 2.39.2 From nobody Fri Dec 19 21:14:37 2025 Received: from mail-lj1-f169.google.com (mail-lj1-f169.google.com [209.85.208.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E7013171676 for ; Tue, 28 May 2024 15:12:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716909180; cv=none; b=ZnLp0QSTFGlflOen05fIYLcOymiGfTtjiMw0AQbQdyKXMroqWaOz4V9e9589yvv48HV+ONigDY1BmbyMed5cabU0hG28vGtxJVi62CFyLeG1pHbpQQ8Heepj7IoJ/Pwa5M+55xAUNiOevmDJ6Gi8kdAArWFpszcPkN6Iwyu0CiQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716909180; c=relaxed/simple; bh=dMj/ll2LOZkWgik8ojL9PklhNmgs0VfbZPpQvfJjzsg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rsXXr5uogoZgoEbf2TdcFwvyasA81Awy0TTL2Bc00xPMzfhBRRE4DsKfjQ8zBbtGWh7auhgpsUvVbEmVDi6prn9eN1/KhJzZufjmvLvBiI2+URY7HQWLr1ZGzR/l6hvh/D8meJ3CF8k/OlhXDrtYw4fhbqWS6c/xrbOAc8Qd0OY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=BzH/12bJ; arc=none smtp.client-ip=209.85.208.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="BzH/12bJ" Received: by mail-lj1-f169.google.com with SMTP id 38308e7fff4ca-2e95a883101so11551561fa.3 for ; Tue, 28 May 2024 08:12:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1716909177; x=1717513977; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zVgeqawgxJKFwBkPgG+fq6QiuKdnJgEdRIwxDEWsZ4Q=; b=BzH/12bJVmuNY1gxJhKva5FRNG0nwKZ2XYOr4Irp+G8Ou/jYfdKeiu2VjF9QeIR1pv timbzErGKzKD4ZUmH2WTeuLl7b4VJgfCW/iC1H/9Q0NITH1VeMA4pdwplHlXxOM+tdxj pTOHN7GwPpFD6LUVVT9ccEDnhHovAYlR4xBD2FJWJj32JnJ/ibdvdkBgsKUykwtyb/Df TSCx7ebBlqLrvAaTFsJOCqiJgX1tjBo8a+EoExAhp9S/5+VC4pjGB0su5W8OJnQ/t9t2 oeiPZFw2H7pSMgIB4M7ry/9pow9m2fp+ZRSEAoDiO/vSUsGrf1gLCPwtb9rSaKkL6Rml SdXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716909177; x=1717513977; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zVgeqawgxJKFwBkPgG+fq6QiuKdnJgEdRIwxDEWsZ4Q=; b=prIHFdV0jOednrTQ8nrNv2bWieGqQBjonre8MuKJn+936niB6v3TNjhIJP5kiLr0i2 sFc51QMF+ooFoowmYbQNWbUxvvONZ9n4dDZF+5hS0HSnMZF34Sf65H15v9dSngb0aliA bvES9pluKxMJh+kjZIfZiGvvyvgZHkplm+/bg0+rmR5XGO8GHC9QxEvTN5lOeJ39TgGG oboRvott30An/3D/87fmcDU5G6O2vw4xrmH+hhMsdlGqSpFhNRPN+f+0rEDUl5XcIPw9 uumCP24NrA1MGUwBTwO5wpoXPEwd2z11K/wGXp4Rp25UPfuFOOnd+aLvIf/PazXDPkFY nyHg== X-Forwarded-Encrypted: i=1; AJvYcCUGh54V+I0cqtVT5U3sv0Bp/7GBpUXLXbs/JZZwCyPABKSuuUEEGNA1YubARqPXKR+dd4vpzTScr0XSwlLI9q+rKkIBxJdnQM/ULDB1 X-Gm-Message-State: AOJu0YxKO9ICYB520Pm0nXq+P80HvBHWv2aknGUqt8o3dd+vkj/wFPUZ 29V/Vf5N9B4GeBdz6xPF4wXQ20FCE6MlMd7N67fo7l6PekzlTiQWAOGNtuV0lGY= X-Google-Smtp-Source: AGHT+IG/weMgFc40n3U80f7qsiGYCL5GCfV3H84Qfrnd5vO+VgM5LRY1gz81bmewik/+pxslmkrhfw== X-Received: by 2002:a2e:9953:0:b0:2e9:843d:63e5 with SMTP id 38308e7fff4ca-2e9843d643bmr22338201fa.0.1716909176965; Tue, 28 May 2024 08:12:56 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4210897c6bbsm146607535e9.27.2024.05.28.08.12.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 May 2024 08:12:56 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH 2/7] riscv: Implement cmpxchg8/16() using Zabha Date: Tue, 28 May 2024 17:10:47 +0200 Message-Id: <20240528151052.313031-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240528151052.313031-1-alexghiti@rivosinc.com> References: <20240528151052.313031-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This adds runtime support for Zabha in cmpxchg8/16 operations. Signed-off-by: Alexandre Ghiti --- arch/riscv/Kconfig | 16 ++++++++++++++++ arch/riscv/Makefile | 10 ++++++++++ arch/riscv/include/asm/cmpxchg.h | 26 ++++++++++++++++++++++++-- 3 files changed, 50 insertions(+), 2 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index b443def70139..05597719bb1c 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -579,6 +579,22 @@ config RISCV_ISA_V_PREEMPTIVE preemption. Enabling this config will result in higher memory consumption due to the allocation of per-task's kernel Vector context. =20 +config TOOLCHAIN_HAS_ZABHA + bool + default y + depends on !64BIT || $(cc-option,-mabi=3Dlp64 -march=3Drv64ima_zabha) + depends on !32BIT || $(cc-option,-mabi=3Dilp32 -march=3Drv32ima_zabha) + depends on AS_HAS_OPTION_ARCH + +config RISCV_ISA_ZABHA + bool "Zabha extension support for atomic byte/half-word operations" + depends on TOOLCHAIN_HAS_ZABHA + default y + help + Adds support to use atomic byte/half-word operations in the kernel. + + If you don't know what to do here, say Y. + config TOOLCHAIN_HAS_ZACAS bool default y diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index d5b60b87998c..f58ac921dece 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -89,6 +89,16 @@ else riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZACAS) :=3D $(riscv-march-y)_zacas endif =20 +# Check if the toolchain supports Zabha +ifdef CONFIG_AS_IS_LLVM +# Support for experimental Zabha was merged in LLVM 19. +KBUILD_CFLAGS +=3D -menable-experimental-extensions +KBUILD_AFLAGS +=3D -menable-experimental-extensions +riscv-march-y :=3D $(riscv-march-y)_zabha1p0 +else +riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZABHA) :=3D $(riscv-march-y)_zabha +endif + # Remove F,D,V from isa string for all. Keep extensions between "fd" and "= v" by # matching non-v and non-multi-letter extensions out with the filter ([^v_= ]*) KBUILD_CFLAGS +=3D -march=3D$(shell echo $(riscv-march-y) | sed -E 's/(rv3= 2ima|rv64ima)fd([^v_]*)v?/\1\2/') diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index 1c50b4821ac8..65de9771078e 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -103,8 +103,14 @@ * indicated by comparing RETURN with OLD. */ =20 -#define __arch_cmpxchg_masked(sc_sfx, prepend, append, r, p, o, n) \ +#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, prepend, append, r, p, o, n= ) \ ({ \ + __label__ zabha, end; \ + \ + asm goto(ALTERNATIVE("nop", "j %[zabha]", 0, \ + RISCV_ISA_EXT_ZABHA, 1) \ + : : : : zabha); \ + \ u32 *__ptr32b =3D (u32 *)((ulong)(p) & ~0x3); \ ulong __s =3D ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ ulong __mask =3D GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ @@ -131,6 +137,17 @@ : "memory"); \ \ r =3D (__typeof__(*(p)))((__retx & __mask) >> __s); \ + goto end; \ + \ +zabha: \ + __asm__ __volatile__ ( \ + prepend \ + " amocas" cas_sfx " %0, %z2, %1\n" \ + append \ + : "+&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ +end: \ }) =20 #define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n= ) \ @@ -175,8 +192,13 @@ end: \ \ switch (sizeof(*__ptr)) { \ case 1: \ + __arch_cmpxchg_masked(sc_sfx, ".b" sc_sfx, \ + prepend, append, \ + __ret, __ptr, __old, __new); \ + break; \ case 2: \ - __arch_cmpxchg_masked(sc_sfx, prepend, append, \ + __arch_cmpxchg_masked(sc_sfx, ".h" sc_sfx, \ + prepend, append, \ __ret, __ptr, __old, __new); \ break; \ case 4: \ --=20 2.39.2 From nobody Fri Dec 19 21:14:37 2025 Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C81BA171E40 for ; Tue, 28 May 2024 15:13:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716909241; cv=none; b=s5p773RM86WR5JhU0Fip/mgAyE2mSV/sbl5beMcScIQ5kI/LzwZ6Sj2aUZYFez4u4mbPTMEgc+H77ZMOs7lanuKLmLqbUizyBEqStrSrI8oYy8sCNuVw+1kmHUeiuQJ/zG68dD2Y9bNQx26hJlB1c/Ub2BVyVjHknXVuTQa5SiE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716909241; c=relaxed/simple; bh=EjieqW/8+JJzLLoBTQFYN8FJzpAcjOBh3yjPb46+kqU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=KqBbjMqPuWz9MByohu0G2TZZYbY6ojG21/tu6f+HqzxIByVoSZq3HpxBlLVOic0/WC0Byl31w9ZE+P2N9zbggJj+G5iFm5hUgkUiGRx6A9AnUq7a3FJVwoHbSsDj+xgVPvl+Ms1PsC+NHkHV2aLH9E/I70cpxTNmqHr7q4yeCkY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=1SbsL9C+; arc=none smtp.client-ip=209.85.128.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="1SbsL9C+" Received: by mail-wm1-f52.google.com with SMTP id 5b1f17b1804b1-42120fc8d1dso4353795e9.2 for ; Tue, 28 May 2024 08:13:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1716909238; x=1717514038; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8+QNuMhUn/mhPs/6E1/sFkjL9Li5Z0ATbah2dvUIKDM=; b=1SbsL9C+aAEmI7DUG6TF8zOSrpOD5d/TBkCYFXAThbsUY9Dl4H0aorxErXDwgaPXRl DfvU4jc//kaQ3fw2I2mS5n98wDQm3XOwcF9hIaHO6z9WCZYi93oVyTtIJxiXmVJPXiwX 2YeVHhxiQlXSEFpx4IYaADDq+VIOpWCcDH7OTJkXWx3rTjKc5fOQoZSmsLy048bUKKgl +PvbB830GO8KKlDZZHFTFShoC3lREJIMS4919Fdvsn2nmd4eIvajLXNbCvC3ZLb8HTev jWX8vwa6E6vOGFYidlE/c9Xxo28Y6Dc1U3BilZmDsgX5X2xBRnCIWNM0ADhoSO6Nushi f1Tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716909238; x=1717514038; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8+QNuMhUn/mhPs/6E1/sFkjL9Li5Z0ATbah2dvUIKDM=; b=JqS4P2bbgeWQdIBlWbfsb/ZgvA6Yz//aVoeyagAuckQio/N8/q9C3oqChqKmUhxGvM 1ut0SjPjGiiQ5eEmiRzDMqeokQ6zWxEL53KI+/RBodNWvqx8Oalt5yTu38vhAcGC8/eW WtNJllcT02cCtsauJyjO559qUUS6VP0y2s/Dfk+uX9n92oXDAdlKjEGQ1g2CIZfev8p9 KWhlveXV6Zl5kqTRi+ED+7rzHVIpbVegR1X/c9Hs4IMYlml8cc2y+s69GmQjW6fIMwir myTNFXPFQ5MDfxcKVM7OSZTCCKbcO1EBeWGVuKCBtsuxfH7syTckDwr6Cyvavs0fZ+x2 Bk0w== X-Forwarded-Encrypted: i=1; AJvYcCW1xqphDle4caV29IHfgQxFyt5e48yvfHQFo8x82uUaZJSfhU6qVS51vE9VWbHbkbvG7SgxSxVG6bII5Pi/2heylUziv7syPUVwYvhG X-Gm-Message-State: AOJu0YxQVDvg4G7KmF/h+Ij46c53bkHCBxsN92eT2Ooe6ELMw/gGWHx8 mRvCg8CS357U8m+lv5kKG/aIpkuR+KEN7MCLn+vkG78EN7nEVPEz2BdTrEL6PzM= X-Google-Smtp-Source: AGHT+IFax4v0kzQv5SDc3SAQTe47rQjW60kRd6Vmso4QAx01vZU4yzSYIPRZPjT4yaPXSVHdqYbyVw== X-Received: by 2002:a5d:558d:0:b0:34d:74cb:c0be with SMTP id ffacd0b85a97d-35526c569cbmr8166449f8f.35.1716909238142; Tue, 28 May 2024 08:13:58 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3557a0908e4sm12094483f8f.63.2024.05.28.08.13.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 May 2024 08:13:57 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH 3/7] riscv: Implement arch_cmpxchg128() using Zacas Date: Tue, 28 May 2024 17:10:48 +0200 Message-Id: <20240528151052.313031-4-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240528151052.313031-1-alexghiti@rivosinc.com> References: <20240528151052.313031-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that Zacas is supported in the kernel, let's use the double word atomic version of amocas to improve the SLUB allocator. Note that we have to select fixed registers, otherwise gcc fails to pick even registers and then produces a reserved encoding which fails to assemble. Signed-off-by: Alexandre Ghiti --- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/cmpxchg.h | 41 ++++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 05597719bb1c..184a9edb04e0 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -101,6 +101,7 @@ config RISCV select GENERIC_VDSO_TIME_NS if HAVE_GENERIC_VDSO select HARDIRQS_SW_RESEND select HAS_IOPORT if MMU + select HAVE_ALIGNED_STRUCT_PAGE select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMALLOC if HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_HUGE_VMAP if MMU && 64BIT && !XIP_KERNEL diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index 65de9771078e..0789fbe38b23 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -242,4 +242,45 @@ end: \ arch_cmpxchg_relaxed((ptr), (o), (n)); \ }) =20 +#ifdef CONFIG_RISCV_ISA_ZACAS + +#define system_has_cmpxchg128() \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS) + +union __u128_halves { + u128 full; + struct { + u64 low, high; + }; +}; + +#define __arch_cmpxchg128(p, o, n, prepend, append) \ +({ \ + __typeof__(*(p)) __o =3D (o); \ + union __u128_halves new =3D { .full =3D (n) }; \ + union __u128_halves old =3D { .full =3D (__o) }; \ + register unsigned long x6 asm ("x6") =3D new.low; \ + register unsigned long x7 asm ("x7") =3D new.high; \ + register unsigned long x28 asm ("x28") =3D old.low; \ + register unsigned long x29 asm ("x29") =3D old.high; \ + \ + __asm__ __volatile__ ( \ + prepend \ + " amocas.q %0, %z2, %1\n" \ + append \ + : "+&r" (x28), "+A" (*(p)) \ + : "rJ" (x6) \ + : "memory"); \ + \ + __o; \ +}) + +#define arch_cmpxchg128(ptr, o, n) \ + __arch_cmpxchg128((ptr), (o), (n), "", " fence rw, rw\n") + +#define arch_cmpxchg128_local(ptr, o, n) \ + __arch_cmpxchg128((ptr), (o), (n), "", "") + +#endif /* CONFIG_RISCV_ISA_ZACAS */ + #endif /* _ASM_RISCV_CMPXCHG_H */ --=20 2.39.2 From nobody Fri Dec 19 21:14:37 2025 Received: from mail-lj1-f171.google.com (mail-lj1-f171.google.com [209.85.208.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2527416FF5E for ; Tue, 28 May 2024 15:15:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716909302; cv=none; b=mpHmuxW/724QC3G/kpFkEOHW1XMVu2AHmMVmVTdJtOxm8iXbGfFNxcyC4dUbpZkwvZZMphmYKzsfeatL2k16aQgrwhGRddXL+PFEfL9Ep0Y0B3NX7c/qYcrsSHxlFgYUkxR7H+KTQEyCRqgwR78uHI0s4ShDQ396f8vpJx1K/+s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716909302; c=relaxed/simple; bh=byVCn7vyyoWVCExOcLktbP9jgpsI9hB8fTXW6dyI5l4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=qbUfbkl9n6ls4n9qUtgEx0yrDC2sI0G6uM0tfC9nCrvPe1ypSXiMDE+lGhNyJ+WQJQeuuviL3tOQRDjNlcYhRL1ti2dcJhY+aVrdjYw8/qYrnD7CHAABW9HOK2e4LUpY5v82GmatxOELDYgNDUeY6V85HbG6C4NZaXnDpgUEe3E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=yntQABQd; arc=none smtp.client-ip=209.85.208.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="yntQABQd" Received: by mail-lj1-f171.google.com with SMTP id 38308e7fff4ca-2e964acff1aso9771961fa.0 for ; Tue, 28 May 2024 08:15:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1716909299; x=1717514099; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QJ9jfwQtVSCqEew5FGQUfrRX44QJ3TNBiU9EdF5jARo=; b=yntQABQd2Un8Wr1ZeQtRfwEySXNs4PXXc/bgKeXzgUEt1lh/x8IbAJLtmuIRWveMrI dKQkyxJ3VIo2a0vAt5uDvuhz6zyhhA2RNlNP/eW+BWUVr5rT0Ww02jxkXbdFQR+Q8yx4 raDEp09sVL4gbBA1DccmNWhyl7gTdZXJvbzruYtfURxWL+hMZc6CcpOf09ku3qW6Cwtu aZoJ1pnA3C1kztX0H8fCuHbLIBtxvZv8Bw/w5xr378I3F4yLwuNSPJzcxSSHH5EMM2Xx IZKSVfSsogKL7bB73qRmhm7xTNzRnMgfVIPLjDylZX4XTRfQ8OesccCwwMZKIXZXlWZ8 93sA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716909299; x=1717514099; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QJ9jfwQtVSCqEew5FGQUfrRX44QJ3TNBiU9EdF5jARo=; b=bfMwXo5Qx7/ap6l586OKNp4ymY3kARobk1ifYQLy49ferfzMPlIxy7t3PyvXEJmzwP DfYhFZE+oKifOhlQwsloJcQqWULbY0RAva1wTHDMWaoA/wYrwzg/VZtNzmN1kaeOxcLb FrF1yJQs9gvvV8XDEvyVGugF2HJcdqMcaVN1Tg28eIjNMmIFVyLT6gPTmUOMoyss/ytj aolXgt71ktMe4pVhKQcSBmBCW8ajwF7PrnBjZ50lWwZ3GF7mNErLM2HFif6FYlfTp1N9 LRte8izEmkpxQzDv82RWpVBBSihlbT3hWqAg3Ddi5SlZIKi82rMyWHuQ3mb90LpftdwP csNg== X-Forwarded-Encrypted: i=1; AJvYcCV3577zn4hT2anrOODMRnwROnvYEqY6BcFRLekK0wq68BUS92ac5naPz+wo41eoLZrSxnXDSYbH0beQ4L2S8be819N8bEHNtxs94PMf X-Gm-Message-State: AOJu0YzLHCChDx5zIY3YdRFzXoGL3XyqUu6eHk1ixYJ6VB34a/TiZuZK OzAdJsRWWFVcqxpqD6AAea8dnmGBljw96RE/MfGEnhueZ8upKb4ibwexetMUXzs= X-Google-Smtp-Source: AGHT+IE9xnl9B6iLJUPGftImcREYfkzUqsxyaCtbYNOO7bdA0+3T64EEnX3IW7R8WIRTpy4sMb+IqA== X-Received: by 2002:a2e:a7c3:0:b0:2e9:546c:544a with SMTP id 38308e7fff4ca-2e95b28ae5emr87639391fa.34.1716909299278; Tue, 28 May 2024 08:14:59 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-421089708f8sm143918715e9.20.2024.05.28.08.14.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 May 2024 08:14:58 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH 4/7] riscv: Implement xchg8/16() using Zabha Date: Tue, 28 May 2024 17:10:49 +0200 Message-Id: <20240528151052.313031-5-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240528151052.313031-1-alexghiti@rivosinc.com> References: <20240528151052.313031-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This adds runtime support for Zabha in xchg8/16() operations. Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/cmpxchg.h | 24 ++++++++++++++++++++++-- arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/kernel/cpufeature.c | 1 + 3 files changed, 24 insertions(+), 2 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index 0789fbe38b23..43696d9e13aa 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -11,8 +11,14 @@ #include #include =20 -#define __arch_xchg_masked(prepend, append, r, p, n) \ +#define __arch_xchg_masked(swap_sfx, prepend, append, r, p, n) \ ({ \ + __label__ zabha, end; \ + \ + asm goto(ALTERNATIVE("nop", "j %[zabha]", 0, \ + RISCV_ISA_EXT_ZABHA, 1) \ + : : : : zabha); \ + \ u32 *__ptr32b =3D (u32 *)((ulong)(p) & ~0x3); \ ulong __s =3D ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ ulong __mask =3D GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ @@ -34,6 +40,17 @@ : "memory"); \ \ r =3D (__typeof__(*(p)))((__retx & __mask) >> __s); \ + goto end; \ + \ +zabha: \ + __asm__ __volatile__ ( \ + prepend \ + " amoswap" swap_sfx " %0, %z2, %1\n" \ + append \ + : "=3D&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ +end: \ }) =20 #define __arch_xchg(sfx, prepend, append, r, p, n) \ @@ -55,8 +72,11 @@ \ switch (sizeof(*__ptr)) { \ case 1: \ + __arch_xchg_masked(".b" sfx, prepend, append, \ + __ret, __ptr, __new); \ + break; \ case 2: \ - __arch_xchg_masked(prepend, append, \ + __arch_xchg_masked(".h" sfx, prepend, append, \ __ret, __ptr, __new); \ break; \ case 4: \ diff --git a/arch/riscv/include/asm/hwcap.h b/arch/riscv/include/asm/hwcap.h index e17d0078a651..f71ddd2ca163 100644 --- a/arch/riscv/include/asm/hwcap.h +++ b/arch/riscv/include/asm/hwcap.h @@ -81,6 +81,7 @@ #define RISCV_ISA_EXT_ZTSO 72 #define RISCV_ISA_EXT_ZACAS 73 #define RISCV_ISA_EXT_XANDESPMU 74 +#define RISCV_ISA_EXT_ZABHA 75 =20 #define RISCV_ISA_EXT_XLINUXENVCFG 127 =20 diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index 3ed2359eae35..8d0f56dd2f53 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -257,6 +257,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] =3D { __RISCV_ISA_EXT_DATA(zihintpause, RISCV_ISA_EXT_ZIHINTPAUSE), __RISCV_ISA_EXT_DATA(zihpm, RISCV_ISA_EXT_ZIHPM), __RISCV_ISA_EXT_DATA(zacas, RISCV_ISA_EXT_ZACAS), + __RISCV_ISA_EXT_DATA(zabha, RISCV_ISA_EXT_ZABHA), __RISCV_ISA_EXT_DATA(zfa, RISCV_ISA_EXT_ZFA), __RISCV_ISA_EXT_DATA(zfh, RISCV_ISA_EXT_ZFH), __RISCV_ISA_EXT_DATA(zfhmin, RISCV_ISA_EXT_ZFHMIN), --=20 2.39.2 From nobody Fri Dec 19 21:14:37 2025 Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 110D6282FA for ; Tue, 28 May 2024 15:16:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716909363; cv=none; b=Q6KDnSktW4FcPZq5u+mAYJg31++68LQDiGh71dw5qocILA4QOXLis1/CPMGdwMkHb3Q2yHsfUOZ4lvRG0Z1/zgzjM4QnoEp2mF6b3YKawECsm0fjwE0qdby8R6P8y0vTbtAzRvw6V73EGxQUsdQvluiAZP25gUAI47QrIXwSgaw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716909363; c=relaxed/simple; bh=zIctHSF8s4ycH2M1Xuee5DajFIDyCPEPGL7Vxs9tOR4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=TDETqNBATFR+vhY/Vt3x6xBU6UzmMIIb0/d5ixk4dDSaJQ9nFP9XGb2aQ71Ya5RobHfp2IZ72QwLRlt3FTU+IsqMkw03sFksgI1Nqw7NEVc0JgEeLSmCvfoezL0dGYUBBw8F6gmwFc93haMhUsZCYNDnZkX9HZmXHDjpYzDfq1Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=B/2cJVLe; arc=none smtp.client-ip=209.85.128.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="B/2cJVLe" Received: by mail-wm1-f47.google.com with SMTP id 5b1f17b1804b1-42121d27861so248885e9.0 for ; Tue, 28 May 2024 08:16:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1716909360; x=1717514160; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=heAbIXv1oMx4SoDoWG1mJ8ALiEwmwn+mI3czPDKQjFY=; b=B/2cJVLeRvjhR6DtdsavZO5f1aH9SugqzUFIN6bfi7wqkO7BbNkCgyuud9c/WcgcFB DNo6vfe3CsksekPdbp+cp00dihZqdQgf0fSLPkKpEJ3su1vbVGAxMtMzsJIGJNWFU7ww zSCw0uMIltcXh/TtfIk+becgE2awNSoxLPMpn5pTJnEsLQod1RIsyuknJPprOsEF1W4X Kt2xWSWD3iTXsS5ORWV3QEEM2/X3/jAGisD+6p2TR1rmyzSxzqrF1kf5ealpxywFmkEK N3mDi5rqgucKvpUu0PNNkZGFM+JC0dKv2V9W3++exx18BTHiwg2a4syRW4GI7R6iUvdr 7zzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716909360; x=1717514160; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=heAbIXv1oMx4SoDoWG1mJ8ALiEwmwn+mI3czPDKQjFY=; b=VMwNx/XeNo/iOTOqr7UAi/hVT8xTZykXJiBxkmB3sgRcmL4/zWyx7NNuy4BQYcOYSc EkIYSWYrlYtbfzPgYHuX7IRF/AqDwUnDP1f6uy8Hpj69fUuAcXz9jbomHzhKgilqKmNB DL8nTe2ZdOZt1cfjLG11Y/KuAfMvoRH+/mlsfa4wkKVynhmuw+Ci9N7Tu2t1b1Wx6zbZ sbfF4EZUgszvUHhTCXVXLXZ2LsmSPFVA+TmPhgCf9RUm5U2tkQgwDY1m8r/XEVeNlSDW 0vdJ6gAwLORc+91BvTMOWsxMRpc89vdJ/6CjiCNGJehgZReADnIhfYGftx9kpiy3x8uE 2LWg== X-Forwarded-Encrypted: i=1; AJvYcCViQQoPiMH3UXfor65XG9GebkxbJxGHnVW8IVB/GDWly9JWfpZgwNxQWuXsjnIIxdkhEsa/DMpckle7K09GG1jvzs2oogh4Vmlzzxc0 X-Gm-Message-State: AOJu0Yw4PJBpOjv9nbKI9oUw7cE2NL/+sCqWR3Dm1J2DVkkangEatwLq pGkVrN8FWOK6biqWTfZu9g2egAq+mGlLHmJhdXlQQlBpt9HLcKqbNWrPK1ElRR4= X-Google-Smtp-Source: AGHT+IFjBUzptjD+yaaIGtCtzbT9/u6NhvlOiOfogXGTAn3lVdgJuuRzm1ZS7bzwllyXZTKIP0WObg== X-Received: by 2002:a05:600c:45cd:b0:420:29dd:84db with SMTP id 5b1f17b1804b1-42108a26c56mr116889895e9.35.1716909360501; Tue, 28 May 2024 08:16:00 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42108970b4esm146682755e9.14.2024.05.28.08.15.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 May 2024 08:16:00 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Guo Ren Subject: [PATCH 5/7] asm-generic: ticket-lock: Reuse arch_spinlock_t of qspinlock Date: Tue, 28 May 2024 17:10:50 +0200 Message-Id: <20240528151052.313031-6-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240528151052.313031-1-alexghiti@rivosinc.com> References: <20240528151052.313031-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Guo Ren The arch_spinlock_t of qspinlock has contained the atomic_t val, which satisfies the ticket-lock requirement. Thus, unify the arch_spinlock_t into qspinlock_types.h. This is the preparation for the next combo spinlock. Reviewed-by: Leonardo Bras Suggested-by: Arnd Bergmann Link: https://lore.kernel.org/linux-riscv/CAK8P3a2rnz9mQqhN6-e0CGUUv9rntREL= Fdxt_weiD7FxH7fkfQ@mail.gmail.com/ Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- include/asm-generic/spinlock.h | 14 +++++++------- include/asm-generic/spinlock_types.h | 12 ++---------- 2 files changed, 9 insertions(+), 17 deletions(-) diff --git a/include/asm-generic/spinlock.h b/include/asm-generic/spinlock.h index 90803a826ba0..4773334ee638 100644 --- a/include/asm-generic/spinlock.h +++ b/include/asm-generic/spinlock.h @@ -32,7 +32,7 @@ =20 static __always_inline void arch_spin_lock(arch_spinlock_t *lock) { - u32 val =3D atomic_fetch_add(1<<16, lock); + u32 val =3D atomic_fetch_add(1<<16, &lock->val); u16 ticket =3D val >> 16; =20 if (ticket =3D=3D (u16)val) @@ -46,31 +46,31 @@ static __always_inline void arch_spin_lock(arch_spinloc= k_t *lock) * have no outstanding writes due to the atomic_fetch_add() the extra * orderings are free. */ - atomic_cond_read_acquire(lock, ticket =3D=3D (u16)VAL); + atomic_cond_read_acquire(&lock->val, ticket =3D=3D (u16)VAL); smp_mb(); } =20 static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) { - u32 old =3D atomic_read(lock); + u32 old =3D atomic_read(&lock->val); =20 if ((old >> 16) !=3D (old & 0xffff)) return false; =20 - return atomic_try_cmpxchg(lock, &old, old + (1<<16)); /* SC, for RCsc */ + return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RC= sc */ } =20 static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) { u16 *ptr =3D (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); - u32 val =3D atomic_read(lock); + u32 val =3D atomic_read(&lock->val); =20 smp_store_release(ptr, (u16)val + 1); } =20 static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) { - u32 val =3D lock.counter; + u32 val =3D lock.val.counter; =20 return ((val >> 16) =3D=3D (val & 0xffff)); } @@ -84,7 +84,7 @@ static __always_inline int arch_spin_is_locked(arch_spinl= ock_t *lock) =20 static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) { - u32 val =3D atomic_read(lock); + u32 val =3D atomic_read(&lock->val); =20 return (s16)((val >> 16) - (val & 0xffff)) > 1; } diff --git a/include/asm-generic/spinlock_types.h b/include/asm-generic/spi= nlock_types.h index 8962bb730945..f534aa5de394 100644 --- a/include/asm-generic/spinlock_types.h +++ b/include/asm-generic/spinlock_types.h @@ -3,15 +3,7 @@ #ifndef __ASM_GENERIC_SPINLOCK_TYPES_H #define __ASM_GENERIC_SPINLOCK_TYPES_H =20 -#include -typedef atomic_t arch_spinlock_t; - -/* - * qrwlock_types depends on arch_spinlock_t, so we must typedef that befor= e the - * include. - */ -#include - -#define __ARCH_SPIN_LOCK_UNLOCKED ATOMIC_INIT(0) +#include +#include =20 #endif /* __ASM_GENERIC_SPINLOCK_TYPES_H */ --=20 2.39.2 From nobody Fri Dec 19 21:14:37 2025 Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3261CF9D6 for ; Tue, 28 May 2024 15:17:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716909425; cv=none; b=OXVfiGeWOSe/fLhItZiF2d0IkqEHlWg8Kt2C9aYG8F2cBrCtsKd0lBM5iyNvLY8LhjDBOGkXvJlmk2l6RJALJJDbm9zScTud147uE+rP2zf6qngfNmOuoZgr5G6C4sOXCFCZK94+LiYYvCWICbCsuSy6f5GtOOU3p2KvulU7Uj8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716909425; c=relaxed/simple; bh=o4ZK/GG6lXAN1cfDXgIpVYZWTFwSck3jVd1HEtW4i7A=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=XbZexfgejFFiFfzlTYpC0Y7sTYrVGrqCiIFLwWOR4Qd+L2jCQ4376kgpjBfbASLtWZAUhjLyb2wjJipoHmR5ZPAZG1iFXK4v9K4Hks2G2DZ/2dlxRdRdRVgGiQ5oCSai9jnDXzIBK2gS9sNPaVLucgtaLQ/FAjjs6bqQy+FgzdY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=HlEkmPBY; arc=none smtp.client-ip=209.85.221.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="HlEkmPBY" Received: by mail-wr1-f46.google.com with SMTP id ffacd0b85a97d-354f14bd80cso599747f8f.1 for ; Tue, 28 May 2024 08:17:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1716909422; x=1717514222; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JpMFH6VmSNUk91jIQ+0kjGx11FiPCz2ltKgHshZWUPg=; b=HlEkmPBYlvrpedzK6tVnunSoPJGo6506m+heu9GCXVFbwWrxdV34Qi6K0FML+A2BF/ CsU9jlUCxMcChAzjUUBXlzmv2b0cwB5TA5xY7tBErLQOMOWYt759orDkcKp+EZeDjCep 8Zrk8foNA8h6YdHMuYqAv3B7niR2uSp5hkq0jHwsnwtl4C1HfoTyD0jtJFNpV3tHxoOl zjS8K+hEbwkiBxE1qv0Gfiv6hyp0YBq7OFE8MVOz1lVFt/rwrP96ScChw4x5uVACanfS +R1xjpryblG3TfpQhBdvekAfKEWj0C8heb2Vt9m0qW+VH1tv+qLekpZwvREfXY0kakS8 Nm+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716909422; x=1717514222; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JpMFH6VmSNUk91jIQ+0kjGx11FiPCz2ltKgHshZWUPg=; b=EfV5C7VT6VsgTP8muXDTcWtp9/U0qbpOBs279+SN21yzTHHstcm/pLtZw69a8MMGij 9p6WPMQoHYDtu045RB08DIXEz+lNWjvDzQ/qurXtYGndmwlrIU+zuSdBpyv0eINGkOY2 rtM5lrrmg1ZzDFb6J/xnKOgxvSBY6Dvzx8Ws/1PznHrZ0yEtLSauktBnwp1Hi4OVQCOR pacAwI2perYd5d50oXp1nOHAwvtYNW++PZU+mYhcmrDt0PJloWSs51XTa7NtROetzfYZ eldZE9isctO3iY0ZtAKhZ5n4QMoaytXm/Fpmznhe2GpQmDp9XyOG/fgB1nppVEB6jq8e TToQ== X-Forwarded-Encrypted: i=1; AJvYcCWcZME/lQFn0J1kwwaebsP61BwuFROnS+pF+MVn7yCLaejee/gJugCBmImAoyz+2prKsZK0jp14c2P9FGjr/Q5sQOnE3GGlrYOstdK/ X-Gm-Message-State: AOJu0Yw0rd5L8kDo/JGAguhkdDFSLW03eZvdSp7uoDUfpEQO7UsCMJpT Jr+F3zjBiiB90bSDQasVnzUXSjJRPWFSHt/s/Ea6IJ/r3/vY25c57mvjqYa5Zpo= X-Google-Smtp-Source: AGHT+IGKRwy2+CQcadHIdXI5RLUj67bi3lAmxmXPhsYNl8Dzwoe0fCZxCfdusybh4z8G1tvkeSzGbg== X-Received: by 2002:a5d:4e43:0:b0:354:f9f0:a7ed with SMTP id ffacd0b85a97d-3552fdfa5d4mr7849836f8f.47.1716909421494; Tue, 28 May 2024 08:17:01 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3564afc3577sm11599649f8f.102.2024.05.28.08.17.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 May 2024 08:17:01 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Guo Ren Subject: [PATCH 6/7] asm-generic: ticket-lock: Add separate ticket-lock.h Date: Tue, 28 May 2024 17:10:51 +0200 Message-Id: <20240528151052.313031-7-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240528151052.313031-1-alexghiti@rivosinc.com> References: <20240528151052.313031-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Guo Ren Add a separate ticket-lock.h to include multiple spinlock versions and select one at compile time or runtime. Reviewed-by: Leonardo Bras Suggested-by: Arnd Bergmann Link: https://lore.kernel.org/linux-riscv/CAK8P3a2rnz9mQqhN6-e0CGUUv9rntREL= Fdxt_weiD7FxH7fkfQ@mail.gmail.com/ Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- include/asm-generic/spinlock.h | 87 +--------------------- include/asm-generic/ticket_spinlock.h | 103 ++++++++++++++++++++++++++ 2 files changed, 104 insertions(+), 86 deletions(-) create mode 100644 include/asm-generic/ticket_spinlock.h diff --git a/include/asm-generic/spinlock.h b/include/asm-generic/spinlock.h index 4773334ee638..970590baf61b 100644 --- a/include/asm-generic/spinlock.h +++ b/include/asm-generic/spinlock.h @@ -1,94 +1,9 @@ /* SPDX-License-Identifier: GPL-2.0 */ =20 -/* - * 'Generic' ticket-lock implementation. - * - * It relies on atomic_fetch_add() having well defined forward progress - * guarantees under contention. If your architecture cannot provide this, = stick - * to a test-and-set lock. - * - * It also relies on atomic_fetch_add() being safe vs smp_store_release() = on a - * sub-word of the value. This is generally true for anything LL/SC althou= gh - * you'd be hard pressed to find anything useful in architecture specifica= tions - * about this. If your architecture cannot do this you might be better off= with - * a test-and-set. - * - * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and = hence - * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along= with - * a full fence after the spin to upgrade the otherwise-RCpc - * atomic_cond_read_acquire(). - * - * The implementation uses smp_cond_load_acquire() to spin, so if the - * architecture has WFE like instructions to sleep instead of poll for word - * modifications be sure to implement that (see ARM64 for example). - * - */ - #ifndef __ASM_GENERIC_SPINLOCK_H #define __ASM_GENERIC_SPINLOCK_H =20 -#include -#include - -static __always_inline void arch_spin_lock(arch_spinlock_t *lock) -{ - u32 val =3D atomic_fetch_add(1<<16, &lock->val); - u16 ticket =3D val >> 16; - - if (ticket =3D=3D (u16)val) - return; - - /* - * atomic_cond_read_acquire() is RCpc, but rather than defining a - * custom cond_read_rcsc() here we just emit a full fence. We only - * need the prior reads before subsequent writes ordering from - * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we - * have no outstanding writes due to the atomic_fetch_add() the extra - * orderings are free. - */ - atomic_cond_read_acquire(&lock->val, ticket =3D=3D (u16)VAL); - smp_mb(); -} - -static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) -{ - u32 old =3D atomic_read(&lock->val); - - if ((old >> 16) !=3D (old & 0xffff)) - return false; - - return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RC= sc */ -} - -static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) -{ - u16 *ptr =3D (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); - u32 val =3D atomic_read(&lock->val); - - smp_store_release(ptr, (u16)val + 1); -} - -static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) -{ - u32 val =3D lock.val.counter; - - return ((val >> 16) =3D=3D (val & 0xffff)); -} - -static __always_inline int arch_spin_is_locked(arch_spinlock_t *lock) -{ - arch_spinlock_t val =3D READ_ONCE(*lock); - - return !arch_spin_value_unlocked(val); -} - -static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) -{ - u32 val =3D atomic_read(&lock->val); - - return (s16)((val >> 16) - (val & 0xffff)) > 1; -} - +#include #include =20 #endif /* __ASM_GENERIC_SPINLOCK_H */ diff --git a/include/asm-generic/ticket_spinlock.h b/include/asm-generic/ti= cket_spinlock.h new file mode 100644 index 000000000000..cfcff22b37b3 --- /dev/null +++ b/include/asm-generic/ticket_spinlock.h @@ -0,0 +1,103 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * 'Generic' ticket-lock implementation. + * + * It relies on atomic_fetch_add() having well defined forward progress + * guarantees under contention. If your architecture cannot provide this, = stick + * to a test-and-set lock. + * + * It also relies on atomic_fetch_add() being safe vs smp_store_release() = on a + * sub-word of the value. This is generally true for anything LL/SC althou= gh + * you'd be hard pressed to find anything useful in architecture specifica= tions + * about this. If your architecture cannot do this you might be better off= with + * a test-and-set. + * + * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and = hence + * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along= with + * a full fence after the spin to upgrade the otherwise-RCpc + * atomic_cond_read_acquire(). + * + * The implementation uses smp_cond_load_acquire() to spin, so if the + * architecture has WFE like instructions to sleep instead of poll for word + * modifications be sure to implement that (see ARM64 for example). + * + */ + +#ifndef __ASM_GENERIC_TICKET_SPINLOCK_H +#define __ASM_GENERIC_TICKET_SPINLOCK_H + +#include +#include + +static __always_inline void ticket_spin_lock(arch_spinlock_t *lock) +{ + u32 val =3D atomic_fetch_add(1<<16, &lock->val); + u16 ticket =3D val >> 16; + + if (ticket =3D=3D (u16)val) + return; + + /* + * atomic_cond_read_acquire() is RCpc, but rather than defining a + * custom cond_read_rcsc() here we just emit a full fence. We only + * need the prior reads before subsequent writes ordering from + * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we + * have no outstanding writes due to the atomic_fetch_add() the extra + * orderings are free. + */ + atomic_cond_read_acquire(&lock->val, ticket =3D=3D (u16)VAL); + smp_mb(); +} + +static __always_inline bool ticket_spin_trylock(arch_spinlock_t *lock) +{ + u32 old =3D atomic_read(&lock->val); + + if ((old >> 16) !=3D (old & 0xffff)) + return false; + + return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RC= sc */ +} + +static __always_inline void ticket_spin_unlock(arch_spinlock_t *lock) +{ + u16 *ptr =3D (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); + u32 val =3D atomic_read(&lock->val); + + smp_store_release(ptr, (u16)val + 1); +} + +static __always_inline int ticket_spin_value_unlocked(arch_spinlock_t lock) +{ + u32 val =3D lock.val.counter; + + return ((val >> 16) =3D=3D (val & 0xffff)); +} + +static __always_inline int ticket_spin_is_locked(arch_spinlock_t *lock) +{ + arch_spinlock_t val =3D READ_ONCE(*lock); + + return !ticket_spin_value_unlocked(val); +} + +static __always_inline int ticket_spin_is_contended(arch_spinlock_t *lock) +{ + u32 val =3D atomic_read(&lock->val); + + return (s16)((val >> 16) - (val & 0xffff)) > 1; +} + +/* + * Remapping spinlock architecture specific functions to the corresponding + * ticket spinlock functions. + */ +#define arch_spin_is_locked(l) ticket_spin_is_locked(l) +#define arch_spin_is_contended(l) ticket_spin_is_contended(l) +#define arch_spin_value_unlocked(l) ticket_spin_value_unlocked(l) +#define arch_spin_lock(l) ticket_spin_lock(l) +#define arch_spin_trylock(l) ticket_spin_trylock(l) +#define arch_spin_unlock(l) ticket_spin_unlock(l) + +#endif /* __ASM_GENERIC_TICKET_SPINLOCK_H */ --=20 2.39.2 From nobody Fri Dec 19 21:14:37 2025 Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6944C17083F for ; Tue, 28 May 2024 15:18:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716909486; cv=none; b=TaDPxxRZmaHKzVR0VPg3tvpNg4wYbeEKFqMkxXBgUmptx/7TwSVyU3FsKuyfo1nZVZ9AW7sZJBApTqDTzlg4I5gvlREpOi81p7uSlXeL4ypcNYOq4Ncn6n0M8npIvia/RZ6QmHC79ZpSK+wT/sw+YSHvM8y9VznBSiarxJ9sJis= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716909486; c=relaxed/simple; bh=b/9Sq8q/PIexMlnwPbVCGIDCKB12/I67c0iNJhhRCS0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Pk3cEPm3Ue5PkzCQeLpiUzZmg3WQcjoW7qJ3+kZtllzOg6EbWOH1ztwQy9UfTeWq8Fqf9FJ+ScKGKt0aByhH0/vxWQoRuAJ4Kgi6BvPLMuWrJl5U3BollaSJ/D3rJOqBuXALQtj5+JLn+HadBN8DPadQGNKpT+aXcYG0Y4OGMCs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=OGEX1ywC; arc=none smtp.client-ip=209.85.128.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="OGEX1ywC" Received: by mail-wm1-f48.google.com with SMTP id 5b1f17b1804b1-4202ca70318so7679955e9.1 for ; Tue, 28 May 2024 08:18:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1716909483; x=1717514283; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Wd7aXtixWAZ5JpuaJhpJ6SfusvaeWHbfBXwMoreMRYM=; b=OGEX1ywCRPn1PD3jpuGhq41pejL88gdHiFt6Bme1MFyJrijMMbzrxg1puzXMEfzmnm XwvMbLp4gdxzyEOv0ivmNNECcQ/2TGph0DddcXmmvABOJec/Sihb2/s3mjMO1tn8TImH qOIZBn2WGpUE2MKqHW22nYZZ0+uujwpEK8GyTUz+wYaFvmE+nLr3RkCZvSxAiPeZl9v3 mX87SpFyv7cdOaar89a8DQgPDph1XARFhVGEpgdXyKpfoPGq4PK6BYX7jfuIXSmgBvp/ hKhvFGcLAHMt82txTsEdhbmELK98fSp1udo1Wd4X78aOTBKNPDkl0+28VUsNf/pY5SN5 fJkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716909483; x=1717514283; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Wd7aXtixWAZ5JpuaJhpJ6SfusvaeWHbfBXwMoreMRYM=; b=NOq+xP9FPMx+4d2E8RhIN1u+cuE08a+MZs7WqzcJYIBhcNIwsYb2F5J+oKAlzREf/K nXWKw8UHNiLalo9HCVNXuygmv4yjPB0iC9VuHRXWdON79JCKuyISHaQ4J9U2MvAfkSbV ePj/M2FAfkSuh9BmHJTUf96QgLkCkzrG3eENjju1pPplWX53aTJ8wz02ATKz4Rwwbc9p 2nuRlUa0GCPCa7hGL3n2oTB4lThX9pVVgX50dUo7S41J+3lpcbQm0DNve9GwMV4+RbHi 1YGiQ86TKOXQrpPSsM/NxYq9FObq17D6fbDvT1xcL+doRB8U+2yQBh3ppTu2vNsIcbHr lVkw== X-Forwarded-Encrypted: i=1; AJvYcCUxbGdADGUtdTO0ZnrovnYjJtV0Vh1z3QasfdI374TjOmtM/brs1N9K/UtqsX6bH4Y3TBuBIhxiZmHD/UWoytVQgr6nMtkTZobjtCSz X-Gm-Message-State: AOJu0YwBcEYosT71U1JWAlIcqSb9JTFNwLYkjnl6uPN02FdOc52UtLl0 AtnGGpPW1zkri0dWWn6+m5kppJil/0/Zzjnu5nm+uBMbfVXT723BfOwzDIF+ov4= X-Google-Smtp-Source: AGHT+IHP86jwpef2VrNTk/F/3m18C35oRI7S2uSuTicrM8OfB1hqEectxv7/nW7zcBNuEDulc3tp2A== X-Received: by 2002:a05:6000:1e4b:b0:34f:7788:37ef with SMTP id ffacd0b85a97d-35526c69eecmr9262367f8f.29.1716909482640; Tue, 28 May 2024 08:18:02 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-357718907ddsm8997697f8f.69.2024.05.28.08.18.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 May 2024 08:18:02 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH 7/7] riscv: Add qspinlock support based on Zabha extension Date: Tue, 28 May 2024 17:10:52 +0200 Message-Id: <20240528151052.313031-8-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240528151052.313031-1-alexghiti@rivosinc.com> References: <20240528151052.313031-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In order to produce a generic kernel, a user can select CONFIG_QUEUED_SPINLOCKS which will fallback at runtime to the ticket spinlock implementation if Zabha is not present. Note that we can't use alternatives here because the discovery of extensions is done too late and we need to start with the qspinlock implementation because the ticket spinlock implementation would pollute the spinlock value, so let's use static keys. This is largely based on Guo's work and Leonardo reviews at [1]. Link: https://lore.kernel.org/linux-riscv/20231225125847.2778638-1-guoren@k= ernel.org/ [1] Signed-off-by: Alexandre Ghiti --- .../locking/queued-spinlocks/arch-support.txt | 2 +- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/Kbuild | 4 +- arch/riscv/include/asm/spinlock.h | 39 +++++++++++++++++++ arch/riscv/kernel/setup.c | 18 +++++++++ include/asm-generic/qspinlock.h | 2 + include/asm-generic/ticket_spinlock.h | 2 + 7 files changed, 66 insertions(+), 2 deletions(-) create mode 100644 arch/riscv/include/asm/spinlock.h diff --git a/Documentation/features/locking/queued-spinlocks/arch-support.t= xt b/Documentation/features/locking/queued-spinlocks/arch-support.txt index 22f2990392ff..cf26042480e2 100644 --- a/Documentation/features/locking/queued-spinlocks/arch-support.txt +++ b/Documentation/features/locking/queued-spinlocks/arch-support.txt @@ -20,7 +20,7 @@ | openrisc: | ok | | parisc: | TODO | | powerpc: | ok | - | riscv: | TODO | + | riscv: | ok | | s390: | TODO | | sh: | TODO | | sparc: | ok | diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 184a9edb04e0..ccf1703edeb9 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -59,6 +59,7 @@ config RISCV select ARCH_SUPPORTS_SHADOW_CALL_STACK if HAVE_SHADOW_CALL_STACK select ARCH_USE_MEMTEST select ARCH_USE_QUEUED_RWLOCKS + select ARCH_USE_QUEUED_SPINLOCKS if TOOLCHAIN_HAS_ZABHA select ARCH_USES_CFI_TRAPS if CFI_CLANG select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH if SMP && MMU select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU diff --git a/arch/riscv/include/asm/Kbuild b/arch/riscv/include/asm/Kbuild index 504f8b7e72d4..ad72f2bd4cc9 100644 --- a/arch/riscv/include/asm/Kbuild +++ b/arch/riscv/include/asm/Kbuild @@ -2,10 +2,12 @@ generic-y +=3D early_ioremap.h generic-y +=3D flat.h generic-y +=3D kvm_para.h +generic-y +=3D mcs_spinlock.h generic-y +=3D parport.h -generic-y +=3D spinlock.h generic-y +=3D spinlock_types.h +generic-y +=3D ticket_spinlock.h generic-y +=3D qrwlock.h generic-y +=3D qrwlock_types.h +generic-y +=3D qspinlock.h generic-y +=3D user.h generic-y +=3D vmlinux.lds.h diff --git a/arch/riscv/include/asm/spinlock.h b/arch/riscv/include/asm/spi= nlock.h new file mode 100644 index 000000000000..e00429ac20ed --- /dev/null +++ b/arch/riscv/include/asm/spinlock.h @@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __ASM_RISCV_SPINLOCK_H +#define __ASM_RISCV_SPINLOCK_H + +#ifdef CONFIG_QUEUED_SPINLOCKS +#define _Q_PENDING_LOOPS (1 << 9) + +#define __no_arch_spinlock_redefine +#include +#include +#include + +DECLARE_STATIC_KEY_TRUE(qspinlock_key); + +#define SPINLOCK_BASE_DECLARE(op, type, type_lock) \ +static __always_inline type arch_spin_##op(type_lock lock) \ +{ \ + if (static_branch_unlikely(&qspinlock_key)) \ + return queued_spin_##op(lock); \ + return ticket_spin_##op(lock); \ +} + +SPINLOCK_BASE_DECLARE(lock, void, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(unlock, void, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(is_locked, int, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(is_contended, int, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(trylock, bool, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(value_unlocked, int, arch_spinlock_t) + +#else + +#include + +#endif + +#include + +#endif /* __ASM_RISCV_SPINLOCK_H */ diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c index 4f73c0ae44b2..31ce75522fd4 100644 --- a/arch/riscv/kernel/setup.c +++ b/arch/riscv/kernel/setup.c @@ -244,6 +244,23 @@ static void __init parse_dtb(void) #endif } =20 +DEFINE_STATIC_KEY_TRUE(qspinlock_key); +EXPORT_SYMBOL(qspinlock_key); + +static void __init riscv_spinlock_init(void) +{ + asm goto(ALTERNATIVE("nop", "j %[qspinlock]", 0, RISCV_ISA_EXT_ZABHA, 1) + : : : : qspinlock); + + static_branch_disable(&qspinlock_key); + pr_info("Ticket spinlock: enabled\n"); + + return; + +qspinlock: + pr_info("Queued spinlock: enabled\n"); +} + extern void __init init_rt_signal_env(void); =20 void __init setup_arch(char **cmdline_p) @@ -295,6 +312,7 @@ void __init setup_arch(char **cmdline_p) riscv_set_dma_cache_alignment(); =20 riscv_user_isa_enable(); + riscv_spinlock_init(); } =20 bool arch_cpu_is_hotpluggable(int cpu) diff --git a/include/asm-generic/qspinlock.h b/include/asm-generic/qspinloc= k.h index 0655aa5b57b2..bf47cca2c375 100644 --- a/include/asm-generic/qspinlock.h +++ b/include/asm-generic/qspinlock.h @@ -136,6 +136,7 @@ static __always_inline bool virt_spin_lock(struct qspin= lock *lock) } #endif =20 +#ifndef __no_arch_spinlock_redefine /* * Remapping spinlock architecture specific functions to the corresponding * queued spinlock functions. @@ -146,5 +147,6 @@ static __always_inline bool virt_spin_lock(struct qspin= lock *lock) #define arch_spin_lock(l) queued_spin_lock(l) #define arch_spin_trylock(l) queued_spin_trylock(l) #define arch_spin_unlock(l) queued_spin_unlock(l) +#endif =20 #endif /* __ASM_GENERIC_QSPINLOCK_H */ diff --git a/include/asm-generic/ticket_spinlock.h b/include/asm-generic/ti= cket_spinlock.h index cfcff22b37b3..325779970d8a 100644 --- a/include/asm-generic/ticket_spinlock.h +++ b/include/asm-generic/ticket_spinlock.h @@ -89,6 +89,7 @@ static __always_inline int ticket_spin_is_contended(arch_= spinlock_t *lock) return (s16)((val >> 16) - (val & 0xffff)) > 1; } =20 +#ifndef __no_arch_spinlock_redefine /* * Remapping spinlock architecture specific functions to the corresponding * ticket spinlock functions. @@ -99,5 +100,6 @@ static __always_inline int ticket_spin_is_contended(arch= _spinlock_t *lock) #define arch_spin_lock(l) ticket_spin_lock(l) #define arch_spin_trylock(l) ticket_spin_trylock(l) #define arch_spin_unlock(l) ticket_spin_unlock(l) +#endif =20 #endif /* __ASM_GENERIC_TICKET_SPINLOCK_H */ --=20 2.39.2