From nobody Thu Dec 18 08:34:04 2025 Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B936711720 for ; Wed, 17 Jul 2024 06:21:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721197265; cv=none; b=XjCR6XepqOy6mQ4O7ngItBwKbvsQdOvdv9Q+g+qr2w+ceMXpkilxGrqlsQjLmTo18noS27sSEqhnVos1UgncDaAK8WZk13QkkcF/ep8iE+9ythL3l7GAizHWHd12sAl2td5wyOBBWmCwTC9c/cMvZ4UcXhruNi/ncj1T7siT0N8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721197265; c=relaxed/simple; bh=SXM4rwRn+leJo0pY0SefkogHqfR8LE/yvz/Y5VWFmfY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=au8BiBsToU1r/Ultth5JXPYaJgUdTj+qAxkj5Z0cNy05PLZoPEmpjXq3633AImJ+lI3tL0OJxqUlCY73bNvCUFm2BcZ7OBKiZpV+UjVaTYMCuKg6fNGpmpUVyNdNEtMjOSc8c/UpTVT5ibFLPzBbc8Z+Ov5IgMQxtRRCHpIKTD0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=NO6C4MFc; arc=none smtp.client-ip=209.85.128.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="NO6C4MFc" Received: by mail-wm1-f52.google.com with SMTP id 5b1f17b1804b1-4266fd39527so47683095e9.1 for ; Tue, 16 Jul 2024 23:21:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1721197261; x=1721802061; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ObJKiPOVe5UThPd0qvfEFz5Dv3ABd72erfyOiynPtd0=; b=NO6C4MFccp6jMYUp14P2O8wo0mBXLQl6LHFIuVweezaHf1jUvJie9NmZVZ0ZlFQi8H wDY18VV5pgJBylc6QQI6pv2EtwmXFCspGDWNt7TfUutbEHRaLi6tKVk4+IA/x09mQ8SD q9gm7kYx5Tlc9vTUvCPdPHChI64mu8vTVM+A5r18hmO0CePM2bGgCc7iUOXwvSvc/5RV V808mT4G9+0wbzBHjzaTFKcZbkKnaR+5NATSw+JpgRp1PsHO0N6k18WYsovFOkYFQgol UegyJfdiNS6B1RKv7Ex6GrE3xMbhpD/CcTFtMxWoRIbsaxHuwi3GSe5KlxnSbSMhmwoF B77Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721197261; x=1721802061; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ObJKiPOVe5UThPd0qvfEFz5Dv3ABd72erfyOiynPtd0=; b=dTbDJRrwdSPrK+zWoq1ECe9pD6mggot1x6dnteH8YYkGf3P1Qth0oMMmqfaCKIMuaI 6a62E5Ycu6sgLAUm4S2e9vXVTyi3MqsPSTBnr7vOYCTNVq6h56+anSQC/nv0xKZ51jqC oPnY5CmwjTSSGUegW5mQ1qxSjh4v4we94juTfoSoyTMLCj2HEpV1gZ6x6NmIzbM6ZjSw Dv9/KrdpmA4sleorKcHhZzRQTqDLEf23lqkGv64wziuMUDic2vZst/VDfuidqntFMluj r8l7BphLxXszfCJ2JgJmtwCCkiQB7ikfKg5Zpp0hmj1QLMyigp1J+cxAl77vTcS+z7Ja adDg== X-Forwarded-Encrypted: i=1; AJvYcCWpuWmbOjFPFPrWELa2D7Nv439kD41SFOqW9Bw1ZE4TBD6XMZgUvVcWEWKW7qDPY7iHpuFToNE2jDSLV0CnKypQNLjR+R+NLWAVJZjR X-Gm-Message-State: AOJu0YwSS+s14awTGTbI39VIcxdStFFxKi0QauFuKRDfsOF6AZOJTlJC BY+Nj09uCRsIYxTsOsYweQZHhQ+41yO23hVAImHrhUzjT2lBkfOkoKxL9qYofEA= X-Google-Smtp-Source: AGHT+IEiqDUUVGHtWHIaIo4kIBhavsk6wpSjfOY4FyjkQmBFXrWWxjPRklD8XBy60kYG9vsl83pQBA== X-Received: by 2002:a5d:6a8c:0:b0:368:37ac:3f95 with SMTP id ffacd0b85a97d-36837ac4476mr188365f8f.31.1721197261060; Tue, 16 Jul 2024 23:21:01 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3680dab3cd1sm10867433f8f.22.2024.07.16.23.21.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 23:21:00 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v3 01/11] riscv: Implement cmpxchg32/64() using Zacas Date: Wed, 17 Jul 2024 08:19:47 +0200 Message-Id: <20240717061957.140712-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240717061957.140712-1-alexghiti@rivosinc.com> References: <20240717061957.140712-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This adds runtime support for Zacas in cmpxchg operations. Signed-off-by: Alexandre Ghiti --- arch/riscv/Kconfig | 17 +++++++++++++++++ arch/riscv/Makefile | 3 +++ arch/riscv/include/asm/cmpxchg.h | 26 +++++++++++++++++++++++--- 3 files changed, 43 insertions(+), 3 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 05ccba8ca33a..1caaedec88c7 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -596,6 +596,23 @@ config RISCV_ISA_V_PREEMPTIVE preemption. Enabling this config will result in higher memory consumption due to the allocation of per-task's kernel Vector context. =20 +config TOOLCHAIN_HAS_ZACAS + bool + default y + depends on !64BIT || $(cc-option,-mabi=3Dlp64 -march=3Drv64ima_zacas) + depends on !32BIT || $(cc-option,-mabi=3Dilp32 -march=3Drv32ima_zacas) + depends on AS_HAS_OPTION_ARCH + +config RISCV_ISA_ZACAS + bool "Zacas extension support for atomic CAS" + depends on TOOLCHAIN_HAS_ZACAS + default y + help + Enable the use of the Zacas ISA-extension to implement kernel atomic + cmpxchg operations when it is detected at boot. + + If you don't know what to do here, say Y. + config TOOLCHAIN_HAS_ZBB bool default y diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index 06de9d365088..9fd13d7a9cc6 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -85,6 +85,9 @@ endif # Check if the toolchain supports Zihintpause extension riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZIHINTPAUSE) :=3D $(riscv-march-y)_zihi= ntpause =20 +# Check if the toolchain supports Zacas +riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZACAS) :=3D $(riscv-march-y)_zacas + # Remove F,D,V from isa string for all. Keep extensions between "fd" and "= v" by # matching non-v and non-multi-letter extensions out with the filter ([^v_= ]*) KBUILD_CFLAGS +=3D -march=3D$(shell echo $(riscv-march-y) | sed -E 's/(rv3= 2ima|rv64ima)fd([^v_]*)v?/\1\2/') diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index 808b4c78462e..5d38153e2f13 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -9,6 +9,7 @@ #include =20 #include +#include =20 #define __arch_xchg_masked(sc_sfx, prepend, append, r, p, n) \ ({ \ @@ -134,21 +135,40 @@ r =3D (__typeof__(*(p)))((__retx & __mask) >> __s); \ }) =20 -#define __arch_cmpxchg(lr_sfx, sc_sfx, prepend, append, r, p, co, o, n) \ +#define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n= ) \ ({ \ + __label__ no_zacas, end; \ register unsigned int __rc; \ \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZACAS)) { \ + asm goto(ALTERNATIVE("j %[no_zacas]", "nop", 0, \ + RISCV_ISA_EXT_ZACAS, 1) \ + : : : : no_zacas); \ + \ + __asm__ __volatile__ ( \ + prepend \ + " amocas" sc_cas_sfx " %0, %z2, %1\n" \ + append \ + : "+&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + goto end; \ + } \ + \ +no_zacas: \ __asm__ __volatile__ ( \ prepend \ "0: lr" lr_sfx " %0, %2\n" \ " bne %0, %z3, 1f\n" \ - " sc" sc_sfx " %1, %z4, %2\n" \ + " sc" sc_cas_sfx " %1, %z4, %2\n" \ " bnez %1, 0b\n" \ append \ "1:\n" \ : "=3D&r" (r), "=3D&r" (__rc), "+A" (*(p)) \ : "rJ" (co o), "rJ" (n) \ : "memory"); \ + \ +end:; \ }) =20 #define _arch_cmpxchg(ptr, old, new, sc_sfx, prepend, append) \ @@ -156,7 +176,7 @@ __typeof__(ptr) __ptr =3D (ptr); \ __typeof__(*(__ptr)) __old =3D (old); \ __typeof__(*(__ptr)) __new =3D (new); \ - __typeof__(*(__ptr)) __ret; \ + __typeof__(*(__ptr)) __ret =3D (old); \ \ switch (sizeof(*__ptr)) { \ case 1: \ --=20 2.39.2 From nobody Thu Dec 18 08:34:04 2025 Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EF9A2171D8 for ; Wed, 17 Jul 2024 06:22:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721197326; cv=none; b=A19YxoO6tEXuML7iCsW5tqTSaInHmJtAp9Xg8ZyEKgx4ryf9s/sUtLTbsLuQAR2iwdNoYfMWjirSC0Dq3ge2FtuuLyIPiYG9yEC0RHMxPI4Q9YO+dztB3+38PrbmY13WnOL0fmL10WRt4OujqUsNL0VdGpb1sDJeQTUWR7aJupc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721197326; c=relaxed/simple; bh=tYSKztlvYXjqrUVGqUlFZcjNQFbhaN4iVt2oyZFJdVA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QE7YqV78+zgM25UuZ8wz1skgtPqM1IxbUOArYJie2vIIt/NhX35t7iaGrwkAsn36mH3qnXvG4QZGHyvj9mGq/zu66V3SyFJk+b1VY0ZzWEm+NnB09AUSJa6AL0lOtP0kIh4h/EigMG0BZSgI8cvGYC2FPC9XQd9DjWC+7EohpbE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=v5D9HJzy; arc=none smtp.client-ip=209.85.128.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="v5D9HJzy" Received: by mail-wm1-f54.google.com with SMTP id 5b1f17b1804b1-426717a2d12so2369525e9.0 for ; Tue, 16 Jul 2024 23:22:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1721197322; x=1721802122; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=T6LJZCR61gmm4dvekAa0Q1TIoLF/ScrG5TtOv9gDZ/w=; b=v5D9HJzykvK6DabPsxgMNqvbW0n51rd+JSbAtFnwceusBsHjz2wdH3U9KMJgg3pTYg 9P5/rvGxRfjFwCgfF95hHSw5MNO1o3vxeI5y7/Q7OsDEjsWa/QCBSlDLLd9MY5C+4fzd tQOzriEYeDbL5YxKpZNKLnu6M9ZaC1TQU2jxFcbbIZ5njJ63yV42BlukYZ6TMnf5xnzz 1e7u0W+ilvjaa6skOURXZN+VWVX/I6DE4TvKHnbqWTHquufyqGhCtDX5w+6bc1kzVeMN Vk+hywQr5W4Ks2X5sZ5soeFKqK5pBFef6h9N8BVcfm5pvn2c1SEsNQb4M+yx4+QGWunA PCgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721197322; x=1721802122; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=T6LJZCR61gmm4dvekAa0Q1TIoLF/ScrG5TtOv9gDZ/w=; b=j5Bbl5GYg8ejziIaHoQqEDj/zZIMui5sRD4EHupmvPPK3Ylrave3RPDDU1QogvzlyX R/lppPNfjOtSZHi6q7jgYaVjW8+Vmml6InxJh9FHRYLxMl7pctjulXNh9FsmljstkGp4 00YwcRN7/MBDWlGn3b2jmWPZUpFkEm31zX5oYnXa9V3px2ikPkCjDo5TtBg1WSI1utVj 7UuBpSIMBeb73bqjWXwC3VY2l05aY9bAxX5MIIh3EP6iZcim8pblC+IO/3E4/eOfujl2 rCMXbfgj9X36bkqycFf39XEKe43d5lgaq6dGY8pYIdhp1ChRHueMvbT9TI2NiSyd4Qcb XC5g== X-Forwarded-Encrypted: i=1; AJvYcCURdtEgx6nqZiBD6cNQo/jYj6a1xXcqWPvGCZx8us+qHaSU8Go+Ri4amY77Vlsd0wkMKozKwRnq1p+nmgeiDlETxQnT+RFYX/QEFjJl X-Gm-Message-State: AOJu0Yw1DAIpQSl3P6O4F+h8Q2AtLE+w2y8jjOI6TBLf0C2hJ+l+yZvg 4ELIZfnZmUt2p07YO6YzN7tDc8h5kV2Rml1c4Gz3CX6fcsdabiHL7G+4/UlNbPg= X-Google-Smtp-Source: AGHT+IFU9J5ZIoNE9rhylPtawNo/bVXV/Y54qi/t2Pf5eJSgSGKg9qXAA/fSZkpR14Z9Af1ks6C3xQ== X-Received: by 2002:a05:600c:1d28:b0:426:6981:1bd with SMTP id 5b1f17b1804b1-427c28fa6a5mr5014165e9.5.1721197322313; Tue, 16 Jul 2024 23:22:02 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4279f23a000sm194474825e9.5.2024.07.16.23.22.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 23:22:01 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v3 02/11] dt-bindings: riscv: Add Zabha ISA extension description Date: Wed, 17 Jul 2024 08:19:48 +0200 Message-Id: <20240717061957.140712-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240717061957.140712-1-alexghiti@rivosinc.com> References: <20240717061957.140712-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add description for the Zabha ISA extension which was ratified in April 2024. Signed-off-by: Alexandre Ghiti Reviewed-by: Guo Ren --- Documentation/devicetree/bindings/riscv/extensions.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Documentation/devicetree/bindings/riscv/extensions.yaml b/Docu= mentation/devicetree/bindings/riscv/extensions.yaml index 468c646247aa..e6436260bdeb 100644 --- a/Documentation/devicetree/bindings/riscv/extensions.yaml +++ b/Documentation/devicetree/bindings/riscv/extensions.yaml @@ -171,6 +171,12 @@ properties: memory types as ratified in the 20191213 version of the privil= eged ISA specification. =20 + - const: zabha + description: | + The Zabha extension for Byte and Halfword Atomic Memory Operat= ions + as ratified at commit 49f49c842ff9 ("Update to Rafified state"= ) of + riscv-zabha. + - const: zacas description: | The Zacas extension for Atomic Compare-and-Swap (CAS) instruct= ions --=20 2.39.2 From nobody Thu Dec 18 08:34:04 2025 Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 51A9D1BF53 for ; Wed, 17 Jul 2024 06:23:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721197387; cv=none; b=gMCpxqhsP+bB9ts9+C4l76OZDvxI8UVYzYDTjtoZwFBxOjRZZRpfjmZ6zcchvZSReOG2Q83ol9mASVTF/QIi6pTA4ulyO1sQiadYfjIzYGTXvCzIrgwT8ZCRKstQjNYDinOkKQWEOrbsSURwfchOi9qmM3IA29Kmh3f6EGqJ8AM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721197387; c=relaxed/simple; bh=bde5DyyiE12OaoL1Ez5kRf1Jyic7KdWN1Z4jecTYxiw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=PWIUJKqRp3kLAUtNK8n4bmCTRzGgB+v4DoFituVgSyJ9mYMl5vfOJ18DxsKZ77Ta2jnrJti48usvhFcJzIikPqsHWE0hXaQdlYkd2JczPDYAOvpYeS/pDTYCuP2zxyhnW1uH1t7Me6w6cWv/CLITomAF8E7p1BMq6JBEzuGsfsM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=z8QF1et9; arc=none smtp.client-ip=209.85.128.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="z8QF1et9" Received: by mail-wm1-f42.google.com with SMTP id 5b1f17b1804b1-426526d30aaso45401685e9.0 for ; Tue, 16 Jul 2024 23:23:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1721197384; x=1721802184; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0QjpPjoLwI+58peQzByots6FWlw3e8eiNriVwaJAk3o=; b=z8QF1et9TcQeCmN7iu6Mx+4JDIhCFTbyXwoTW/WlV4PeyL8mygeMgxKkbqvShnttHk 89j8FIpxMw29KoQeA6d3OgOgrCNV63QmzHwtzjUh8dVe1O1KX/8l3ZD6HB3Et2mp/qDk zOWx/5ksllZrdQvIcpWAH+TtiGm6vRdI/cTEAIbfIRv7/QgTHcoUgHLqyVPlxLil9Rlg HHPHoRSWiW1AsuDj14pVPIZ5cB8n7ySiZQhT5RZ5E7HlnLhB3wIPsPCUCHQT69ZiAnPa 5FjOkn+Cr+KSGvfRcio3CDMnLgY0uGozbkaI8FfnFPwB1Zw3KRZdPDu8Jicute4Ui3iu p91w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721197384; x=1721802184; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0QjpPjoLwI+58peQzByots6FWlw3e8eiNriVwaJAk3o=; b=INvdFnTst/h9INjuftt/qE6gepftsM8RLngA4cHDqd7qQIAH0LV1+37o4XYEnNPZ5E mTRhwS8MSb6XHDz48LzxIPnopGZIIxa0hZiuzFk0ZKVh+VmkXEeDFHLc9wrrY7cbB5bn 8N8MJrxL9Y4WaWoP+kmHBsKmx0x6T8n/IvklNKqIWF4YEJBSOa5cgyQ7YPjH90hHs6k7 PSXJZS9grXtpsn649jwXqfkGU9njUxWo5JimlAd0uwfrD6nt82vmbBw1kmTjELxRmvFX o9mcVgVm+varm1qh8sn+Wb8wLtG6qlA3EcVdDiX8HcHDfSQbwESFDHMPO0Vtiw6Aa33J hvQw== X-Forwarded-Encrypted: i=1; AJvYcCUmdv//2iAP1oGjOGX1B9D5diGj8y6DDqlHIIGDgS9t8C7yMZy+4vxV5pRj8W8eYgTQQss45+malrMVy8WRn4f8S8jKOq87ARL281u5 X-Gm-Message-State: AOJu0Yz05cU+gl5OMcn96EN9Ol42AE2FeFXO1mCW61230GyVMGkdL/iq 1/f+rX5XdTbCPGEGSqvM+89d2xTcwXsMEg47hftClgOS2NACT06INzz/xt2+bhI= X-Google-Smtp-Source: AGHT+IFPLb6pDwn4ZT2PDYb1qbrSJD7FUpoL0IeRpIU13kaHx3oluVLPGx9Y3b3221ag/XCVCbja3g== X-Received: by 2002:a05:600c:3592:b0:426:5dca:50a with SMTP id 5b1f17b1804b1-427c2ce6539mr4895335e9.21.1721197383614; Tue, 16 Jul 2024 23:23:03 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3680db03e41sm10773043f8f.98.2024.07.16.23.23.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 23:23:03 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v3 03/11] riscv: Implement cmpxchg8/16() using Zabha Date: Wed, 17 Jul 2024 08:19:49 +0200 Message-Id: <20240717061957.140712-4-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240717061957.140712-1-alexghiti@rivosinc.com> References: <20240717061957.140712-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This adds runtime support for Zabha in cmpxchg8/16() operations. Note that in the absence of Zacas support in the toolchain, CAS instructions from Zabha won't be used. Signed-off-by: Alexandre Ghiti --- arch/riscv/Kconfig | 17 ++++++++++++++++ arch/riscv/Makefile | 3 +++ arch/riscv/include/asm/cmpxchg.h | 33 ++++++++++++++++++++++++++++++-- arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/kernel/cpufeature.c | 1 + 5 files changed, 53 insertions(+), 2 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 1caaedec88c7..d3b0f92f92da 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -596,6 +596,23 @@ config RISCV_ISA_V_PREEMPTIVE preemption. Enabling this config will result in higher memory consumption due to the allocation of per-task's kernel Vector context. =20 +config TOOLCHAIN_HAS_ZABHA + bool + default y + depends on !64BIT || $(cc-option,-mabi=3Dlp64 -march=3Drv64ima_zabha) + depends on !32BIT || $(cc-option,-mabi=3Dilp32 -march=3Drv32ima_zabha) + depends on AS_HAS_OPTION_ARCH + +config RISCV_ISA_ZABHA + bool "Zabha extension support for atomic byte/halfword operations" + depends on TOOLCHAIN_HAS_ZABHA + default y + help + Enable the use of the Zabha ISA-extension to implement kernel + byte/halfword atomic memory operations when it is detected at boot. + + If you don't know what to do here, say Y. + config TOOLCHAIN_HAS_ZACAS bool default y diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index 9fd13d7a9cc6..78dcaaeebf4e 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -88,6 +88,9 @@ riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZIHINTPAUSE) :=3D $(ri= scv-march-y)_zihintpause # Check if the toolchain supports Zacas riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZACAS) :=3D $(riscv-march-y)_zacas =20 +# Check if the toolchain supports Zabha +riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZABHA) :=3D $(riscv-march-y)_zabha + # Remove F,D,V from isa string for all. Keep extensions between "fd" and "= v" by # matching non-v and non-multi-letter extensions out with the filter ([^v_= ]*) KBUILD_CFLAGS +=3D -march=3D$(shell echo $(riscv-march-y) | sed -E 's/(rv3= 2ima|rv64ima)fd([^v_]*)v?/\1\2/') diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index 5d38153e2f13..c86722a101d0 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -105,8 +105,30 @@ * indicated by comparing RETURN with OLD. */ =20 -#define __arch_cmpxchg_masked(sc_sfx, prepend, append, r, p, o, n) \ +#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, prepend, append, r, p, o, n= ) \ ({ \ + __label__ no_zabha_zacas, end; \ + \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && \ + IS_ENABLED(CONFIG_RISCV_ISA_ZACAS)) { \ + asm goto(ALTERNATIVE("j %[no_zabha_zacas]", "nop", 0, \ + RISCV_ISA_EXT_ZABHA, 1) \ + : : : : no_zabha_zacas); \ + asm goto(ALTERNATIVE("j %[no_zabha_zacas]", "nop", 0, \ + RISCV_ISA_EXT_ZACAS, 1) \ + : : : : no_zabha_zacas); \ + \ + __asm__ __volatile__ ( \ + prepend \ + " amocas" cas_sfx " %0, %z2, %1\n" \ + append \ + : "+&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + goto end; \ + } \ + \ +no_zabha_zacas:; \ u32 *__ptr32b =3D (u32 *)((ulong)(p) & ~0x3); \ ulong __s =3D ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ ulong __mask =3D GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ @@ -133,6 +155,8 @@ : "memory"); \ \ r =3D (__typeof__(*(p)))((__retx & __mask) >> __s); \ + \ +end:; \ }) =20 #define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n= ) \ @@ -180,8 +204,13 @@ end:; \ \ switch (sizeof(*__ptr)) { \ case 1: \ + __arch_cmpxchg_masked(sc_sfx, ".b" sc_sfx, \ + prepend, append, \ + __ret, __ptr, __old, __new); \ + break; \ case 2: \ - __arch_cmpxchg_masked(sc_sfx, prepend, append, \ + __arch_cmpxchg_masked(sc_sfx, ".h" sc_sfx, \ + prepend, append, \ __ret, __ptr, __old, __new); \ break; \ case 4: \ diff --git a/arch/riscv/include/asm/hwcap.h b/arch/riscv/include/asm/hwcap.h index e17d0078a651..f71ddd2ca163 100644 --- a/arch/riscv/include/asm/hwcap.h +++ b/arch/riscv/include/asm/hwcap.h @@ -81,6 +81,7 @@ #define RISCV_ISA_EXT_ZTSO 72 #define RISCV_ISA_EXT_ZACAS 73 #define RISCV_ISA_EXT_XANDESPMU 74 +#define RISCV_ISA_EXT_ZABHA 75 =20 #define RISCV_ISA_EXT_XLINUXENVCFG 127 =20 diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index 5ef48cb20ee1..c125d82c894b 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -257,6 +257,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] =3D { __RISCV_ISA_EXT_DATA(zihintpause, RISCV_ISA_EXT_ZIHINTPAUSE), __RISCV_ISA_EXT_DATA(zihpm, RISCV_ISA_EXT_ZIHPM), __RISCV_ISA_EXT_DATA(zacas, RISCV_ISA_EXT_ZACAS), + __RISCV_ISA_EXT_DATA(zabha, RISCV_ISA_EXT_ZABHA), __RISCV_ISA_EXT_DATA(zfa, RISCV_ISA_EXT_ZFA), __RISCV_ISA_EXT_DATA(zfh, RISCV_ISA_EXT_ZFH), __RISCV_ISA_EXT_DATA(zfhmin, RISCV_ISA_EXT_ZFHMIN), --=20 2.39.2 From nobody Thu Dec 18 08:34:04 2025 Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8AD24125DE for ; Wed, 17 Jul 2024 06:24:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721197448; cv=none; b=a7w+SOyTv1N1mErifRPbcSey+GVYPpWzl+L0v1NZQ/2wxubu2kV7vIsyAibX6aX257oA1P8WIgMcF/4ABGoTQUuPh3FLGlTrbFjsEfGevAVcBB26XLEqjougbPsf/ENttymFuWfKGNkOinRh78P5rZAVkujNtcAcB0cChrUe9kE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721197448; c=relaxed/simple; bh=W3ZbUrXKMQk615ONwljbbIROXFCdQYzyCE9meMsyGpU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=aZVqkQsbCtuCvO4YCKzJdLKkQ1VM+oeyLFLvxd5eJOrhBzb/CzepTazYrQb+EXQaQJa3xfJt6/H/vX++UlB47ejpcAw5uNV2sPP7OTyKS45ORMOpq34hRYCabzyj3Hqn8tUGrS0/vkiZnMx6p3FIHhyJxt+pw/Ud0Xdo0nR2ieA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=CfRoQnke; arc=none smtp.client-ip=209.85.128.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="CfRoQnke" Received: by mail-wm1-f44.google.com with SMTP id 5b1f17b1804b1-4266f535e82so43766685e9.1 for ; Tue, 16 Jul 2024 23:24:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1721197445; x=1721802245; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ds1f3nvde/2kobU8ZNZg62T2Qi5mJIAm9dc1xPbfhas=; b=CfRoQnkeRpoGzFiYfi6InQC0w6ja+pTl8/iannG30wLicWgyIvZwjwgySEkmIg/18y NuRpsbsM5X2vjutJtdsDadV5iZol124BfslIWUUhsBVVMBUVJDl+GuHHlxviIqCRCaIn nbl/xDU7oA2WfW8Nf6ugfACV76jmV3FUcauXrFTy5pj+ZJesHqtVp4BBHvv0BB5MJVYs C9ToNTUH1DimQjmb6g3Rb+lbm5kG0yje99R/z1SfTKEGG7U1KtBe5fps3pmmu7A8wakk 0RTw0mnjjZSyyw/QgkPjP8MZE64gNFGE+jDr+EF5ZfF5yRqqtYCXk3M1k521Z+jFrRgA e2tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721197445; x=1721802245; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ds1f3nvde/2kobU8ZNZg62T2Qi5mJIAm9dc1xPbfhas=; b=ofB1UHgxX5O9Wba/lSkFY4/JNqRlwwOYzQmWjFZLoReLmeXFYxIo8L/iALDEYH1AqS kB0HGLl0T6DgD8a5kZTClJTV8DxFiPLAHA7uHDXxxxpBILgehPMRjBjvV8V74QeHeL8W 7EMKe/vvMkPk2qsUykx2QZHOc9jOf9kZQcE8QMLcp2/HKL7t7fIYog7Sr8a/oML2xJoq MUn4uEUmTvpgMwWNsht7g1pHB+me98rWs4ltPb2xozBHh4Juw4bcrpsiYSyw05QI0KSf 4LWKgHq/5kgNiiytiKdl6IkYTYcgsaOAeF8oorYXGCrZJ3caD1W1pkW8nzyQ5PeaWOFk nKcg== X-Forwarded-Encrypted: i=1; AJvYcCWf3goDzpA8CJi7OJQ6wV+OrFeeHQXRy6DTTshlee4mrRYUDRkfGIf4eCXqElHrJ4MGhcZUyOXyLBCHTYdJ7v2PsHIB/UIVOt1R0DIS X-Gm-Message-State: AOJu0Yy7Do3zazareQ+DVPkodXHBVnhKtiUo+bvgOiqFYgd619NWvY0S wIOGX4dOn5SluHVympoapQRHAEndtgjpvz2lpiDRvQArskIvmoxXSw4Sm0i8nFs= X-Google-Smtp-Source: AGHT+IEIXpdS+STkowwy/Wm5vUbJoSbRCAM24AYCj9d6c8yOJneivGu6HLD2duOFWquiyMpaznkmeg== X-Received: by 2002:a5d:6b8d:0:b0:368:75:26e5 with SMTP id ffacd0b85a97d-368316003e7mr683121f8f.1.1721197444910; Tue, 16 Jul 2024 23:24:04 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3680dafba03sm10826606f8f.76.2024.07.16.23.24.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 23:24:04 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrea Parri Subject: [PATCH v3 04/11] riscv: Improve zacas fully-ordered cmpxchg() Date: Wed, 17 Jul 2024 08:19:50 +0200 Message-Id: <20240717061957.140712-5-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240717061957.140712-1-alexghiti@rivosinc.com> References: <20240717061957.140712-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The current fully-ordered cmpxchgXX() implementation results in: amocas.X.rl a5,a4,(s1) fence rw,rw This provides enough sync but we can actually use the following better mapping instead: amocas.X.aqrl a5,a4,(s1) Suggested-by: Andrea Parri Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/cmpxchg.h | 71 ++++++++++++++++++++------------ 1 file changed, 44 insertions(+), 27 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index c86722a101d0..97b24da38897 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -105,7 +105,10 @@ * indicated by comparing RETURN with OLD. */ =20 -#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, prepend, append, r, p, o, n= ) \ +#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + r, p, o, n) \ ({ \ __label__ no_zabha_zacas, end; \ \ @@ -119,9 +122,9 @@ : : : : no_zabha_zacas); \ \ __asm__ __volatile__ ( \ - prepend \ + cas_prepend \ " amocas" cas_sfx " %0, %z2, %1\n" \ - append \ + cas_append \ : "+&r" (r), "+A" (*(p)) \ : "rJ" (n) \ : "memory"); \ @@ -139,7 +142,7 @@ no_zabha_zacas:; \ ulong __rc; \ \ __asm__ __volatile__ ( \ - prepend \ + sc_prepend \ "0: lr.w %0, %2\n" \ " and %1, %0, %z5\n" \ " bne %1, %z3, 1f\n" \ @@ -147,7 +150,7 @@ no_zabha_zacas:; \ " or %1, %1, %z4\n" \ " sc.w" sc_sfx " %1, %1, %2\n" \ " bnez %1, 0b\n" \ - append \ + sc_append \ "1:\n" \ : "=3D&r" (__retx), "=3D&r" (__rc), "+A" (*(__ptr32b)) \ : "rJ" ((long)__oldx), "rJ" (__newx), \ @@ -159,7 +162,10 @@ no_zabha_zacas:; \ end:; \ }) =20 -#define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n= ) \ +#define __arch_cmpxchg(lr_sfx, sc_sfx, cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + r, p, co, o, n) \ ({ \ __label__ no_zacas, end; \ register unsigned int __rc; \ @@ -170,9 +176,9 @@ end:; \ : : : : no_zacas); \ \ __asm__ __volatile__ ( \ - prepend \ - " amocas" sc_cas_sfx " %0, %z2, %1\n" \ - append \ + cas_prepend \ + " amocas" cas_sfx " %0, %z2, %1\n" \ + cas_append \ : "+&r" (r), "+A" (*(p)) \ : "rJ" (n) \ : "memory"); \ @@ -181,12 +187,12 @@ end:; \ \ no_zacas: \ __asm__ __volatile__ ( \ - prepend \ + sc_prepend \ "0: lr" lr_sfx " %0, %2\n" \ " bne %0, %z3, 1f\n" \ - " sc" sc_cas_sfx " %1, %z4, %2\n" \ + " sc" sc_sfx " %1, %z4, %2\n" \ " bnez %1, 0b\n" \ - append \ + sc_append \ "1:\n" \ : "=3D&r" (r), "=3D&r" (__rc), "+A" (*(p)) \ : "rJ" (co o), "rJ" (n) \ @@ -195,7 +201,9 @@ no_zacas: \ end:; \ }) =20 -#define _arch_cmpxchg(ptr, old, new, sc_sfx, prepend, append) \ +#define _arch_cmpxchg(ptr, old, new, sc_sfx, cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append) \ ({ \ __typeof__(ptr) __ptr =3D (ptr); \ __typeof__(*(__ptr)) __old =3D (old); \ @@ -204,22 +212,28 @@ end:; \ \ switch (sizeof(*__ptr)) { \ case 1: \ - __arch_cmpxchg_masked(sc_sfx, ".b" sc_sfx, \ - prepend, append, \ - __ret, __ptr, __old, __new); \ + __arch_cmpxchg_masked(sc_sfx, ".b" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, __old, __new); \ break; \ case 2: \ - __arch_cmpxchg_masked(sc_sfx, ".h" sc_sfx, \ - prepend, append, \ - __ret, __ptr, __old, __new); \ + __arch_cmpxchg_masked(sc_sfx, ".h" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, __old, __new); \ break; \ case 4: \ - __arch_cmpxchg(".w", ".w" sc_sfx, prepend, append, \ - __ret, __ptr, (long), __old, __new); \ + __arch_cmpxchg(".w", ".w" sc_sfx, ".w" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, (long), __old, __new); \ break; \ case 8: \ - __arch_cmpxchg(".d", ".d" sc_sfx, prepend, append, \ - __ret, __ptr, /**/, __old, __new); \ + __arch_cmpxchg(".d", ".d" sc_sfx, ".d" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, /**/, __old, __new); \ break; \ default: \ BUILD_BUG(); \ @@ -228,16 +242,19 @@ end:; \ }) =20 #define arch_cmpxchg_relaxed(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", "", "") + _arch_cmpxchg((ptr), (o), (n), "", "", "", "", "", "") =20 #define arch_cmpxchg_acquire(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", "", RISCV_ACQUIRE_BARRIER) + _arch_cmpxchg((ptr), (o), (n), "", "", \ + "", RISCV_ACQUIRE_BARRIER, "", RISCV_ACQUIRE_BARRIER) =20 #define arch_cmpxchg_release(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", RISCV_RELEASE_BARRIER, "") + _arch_cmpxchg((ptr), (o), (n), "", "", \ + RISCV_RELEASE_BARRIER, "", RISCV_RELEASE_BARRIER, "") =20 #define arch_cmpxchg(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), ".rl", "", " fence rw, rw\n") + _arch_cmpxchg((ptr), (o), (n), ".rl", ".aqrl", \ + "", RISCV_FULL_BARRIER, "", "") =20 #define arch_cmpxchg_local(ptr, o, n) \ arch_cmpxchg_relaxed((ptr), (o), (n)) --=20 2.39.2 From nobody Thu Dec 18 08:34:04 2025 Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 25BAD125DE for ; Wed, 17 Jul 2024 06:25:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721197510; cv=none; b=o9Dq/L5fbo+5XYtSi510AqGIbV9j4/pPCP2OkJcCJi5kqKJNQ10F8Opv89eW6sh9SxU1dCIfnolzFxqjIDs7xVUNgcyK3cPRr7RNA/1dAJXCrV6/MtZAeAWqtWY8A9LsABs1SYdV031uab/Fgdj5oxj8KKNXBaaUVNs+qb5QXkM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721197510; c=relaxed/simple; bh=3zBJRLkrzJHSNuZAH1F+78p9BTEz4I4epc4BJVRS110=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Y28/YP4h+v4r9kjKFofVfe8bVslGdWSnsFkaHrRoli8C9Gd37hrEkJZ3OzOUaTq0CaNduTAUCUuXhm8CRJX49c34dyjnWrfOyDpxs6TAGqE0yamiq0FkLPTu/eBQKuxzLpliGrquC4LETsO1751buHU7fU4QrfnU3wu4+Qq0VDs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=Z8u7SVU+; arc=none smtp.client-ip=209.85.128.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="Z8u7SVU+" Received: by mail-wm1-f47.google.com with SMTP id 5b1f17b1804b1-42660b8dd27so43722035e9.3 for ; Tue, 16 Jul 2024 23:25:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1721197506; x=1721802306; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xQbGd0qu6ssbesu7Zjwn0myIi2A+oWSgEr0KZDNP1pQ=; b=Z8u7SVU+9/ifkGY9+pBpuJd2k664kK38R+eCRwtf6S/rd+EPbFXqEBedrvmhJp5qra n5o05/9r3FuV64FAW4Byt/h4o1BlK5gM1mk0aC/OkNwbVJxiSdBP7bQwF690a/TegAOQ KVkuBRVPSat0xrc0f/4uRpsggwcfpvhetqds7GtGu90gzexZMOfpT2Fs5GchfFb2Z6RE CnwBqMGYmCPizMz5PR6VZGsIcrmsWri38usivn2PRriBnRPhEhmYSP7b0b0Ttz2PkOdR Iy6NGVm2E4Claushy76Q6NvyR9LtDR9gzIkuZmJ0t/2O3giWlcxJ/INUbh+tYMq1yVS7 s1zA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721197506; x=1721802306; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xQbGd0qu6ssbesu7Zjwn0myIi2A+oWSgEr0KZDNP1pQ=; b=Ad7j4d0nkiUci096JJZgcXTOq/LuSBKIiCYjyX+ae8Aqt9HVKGQ8lIV8/cLiRv9XEt yOYtcLapleYlZijWx+75aFiiIP05qSfCAqrIvosscjXB3xIPqjAYXt2CIj8Jt7CJAG9N TKeYgRviD7bPWg+CuzxeUzWfmS/wI3yXsLKc403eKAJmwzYWshpFwZxqmM/1DQ9Nzg0g PIukAaX1NRjkW86EvCKBpxMfY1Vw1NnZfxfXxcw+bVIhwuq2CHLP4jycQou9wTP4aSKF JZaJV54yUOTkwAP/s9eOREAQHteSl1bSA3ememAy1fEGncxtt9ltC5xsQCsFH4iKnpXW 9LGw== X-Forwarded-Encrypted: i=1; AJvYcCVYCJC0Pyg0IT73WSTf2gXej4g019KSJj85A3CNweYHh3UkNEK4sL6tCFwcyoArvNwAG/vRs6cD9rw0U+8ZmWtu498UouiWsyZk/Duy X-Gm-Message-State: AOJu0YxOUChmUS3HxCaGAl89QjmRDHh9ZbXeiyUZlzjcw94wmg1b9Pwg x7QVLgeqiqzgpKddCBoFOISZWnwlutGpS7wfZJj18yVP9Bz+GiXOfhYZ+YX09ow= X-Google-Smtp-Source: AGHT+IEFDv6V5tFWhauDPHJiZCgq6FMlgRvbm1hC+8pd4MPLkSbI6qpbIzPF0VaZZ69mfVtEYemITg== X-Received: by 2002:a05:600c:4eca:b0:426:549c:294c with SMTP id 5b1f17b1804b1-427c2d06e43mr5639015e9.35.1721197506505; Tue, 16 Jul 2024 23:25:06 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-427a5ef4617sm155918965e9.41.2024.07.16.23.25.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 23:25:06 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v3 05/11] riscv: Implement arch_cmpxchg128() using Zacas Date: Wed, 17 Jul 2024 08:19:51 +0200 Message-Id: <20240717061957.140712-6-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240717061957.140712-1-alexghiti@rivosinc.com> References: <20240717061957.140712-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that Zacas is supported in the kernel, let's use the double word atomic version of amocas to improve the SLUB allocator. Note that we have to select fixed registers, otherwise gcc fails to pick even registers and then produces a reserved encoding which fails to assemble. Signed-off-by: Alexandre Ghiti --- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/cmpxchg.h | 39 ++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index d3b0f92f92da..0bbaec0444d0 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -104,6 +104,7 @@ config RISCV select GENERIC_VDSO_TIME_NS if HAVE_GENERIC_VDSO select HARDIRQS_SW_RESEND select HAS_IOPORT if MMU + select HAVE_ALIGNED_STRUCT_PAGE select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMALLOC if HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_HUGE_VMAP if MMU && 64BIT diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index 97b24da38897..608d98522557 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -289,4 +289,43 @@ end:; \ arch_cmpxchg_release((ptr), (o), (n)); \ }) =20 +#ifdef CONFIG_RISCV_ISA_ZACAS + +#define system_has_cmpxchg128() \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS) + +union __u128_halves { + u128 full; + struct { + u64 low, high; + }; +}; + +#define __arch_cmpxchg128(p, o, n, cas_sfx) \ +({ \ + __typeof__(*(p)) __o =3D (o); \ + union __u128_halves __hn =3D { .full =3D (n) }; \ + union __u128_halves __ho =3D { .full =3D (__o) }; \ + register unsigned long x6 asm ("x6") =3D __hn.low; \ + register unsigned long x7 asm ("x7") =3D __hn.high; \ + register unsigned long x28 asm ("x28") =3D __ho.low; \ + register unsigned long x29 asm ("x29") =3D __ho.high; \ + \ + __asm__ __volatile__ ( \ + " amocas.q" cas_sfx " %0, %z3, %2" \ + : "+&r" (x28), "+&r" (x29), "+A" (*(p)) \ + : "rJ" (x6), "rJ" (x7) \ + : "memory"); \ + \ + ((u128)x29 << 64) | x28; \ +}) + +#define arch_cmpxchg128(ptr, o, n) \ + __arch_cmpxchg128((ptr), (o), (n), ".aqrl") + +#define arch_cmpxchg128_local(ptr, o, n) \ + __arch_cmpxchg128((ptr), (o), (n), "") + +#endif /* CONFIG_RISCV_ISA_ZACAS */ + #endif /* _ASM_RISCV_CMPXCHG_H */ --=20 2.39.2 From nobody Thu Dec 18 08:34:04 2025 Received: from mail-wr1-f50.google.com (mail-wr1-f50.google.com [209.85.221.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 61BE014A96 for ; Wed, 17 Jul 2024 06:26:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721197571; cv=none; b=MvhZrvGqA/SKto+EKYSZKvMPHNOwPY6XI0qaNRvI7OOwD7q5QvqVGjEurY9JY4YPfVg6kgtRx06trOQ4lXipSw7u9TcTUNw4RNguP1Eh/gza3EprmUrNs3uVLpBUHkPiN7beYUp9WHkmcX9J30AbssvH22v5GpKIxNqgA9gZo0s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721197571; c=relaxed/simple; bh=B1CnHSwQzyKmmdw+P7nNrDoltmKChpydt1ud39gLqxQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gbCvhvqOp+pSBjaiIDht1YCuRC/AH9p/89+xAdWEeBrOg11EPbEB2JZUQOB8x/hYqk8IC/5XOuts4Hb1I1fXUbuchLhlRziP492I0Gdyn+lY65PPppVr11E5UN3oOdUFgiln1H7mp/nwn50zqhs49WFp58DqiKExldIUNE0rask= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=FQgOsYRw; arc=none smtp.client-ip=209.85.221.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="FQgOsYRw" Received: by mail-wr1-f50.google.com with SMTP id ffacd0b85a97d-367ab50a07aso4010931f8f.0 for ; Tue, 16 Jul 2024 23:26:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1721197568; x=1721802368; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3zMQJJ1+QlmFm418V6AGBGbWQY3rNoVY0h/UyfCjPDE=; b=FQgOsYRwvLW0Cu4ZvNaG5VIIAj9nXfg+4Vxz5MVKLHJGXYneZ5jZ09G6DFlGlwZQQO Y96ai4rvo3iqFPOamN2ubA4HbLsuGEFv1KylQ8xJo9WnOYi4zVCev7mqAgDEY9Y9vOi1 T6WDpjCbV1gNEWZhwtkF6CGhEZQZbp9menI/4ieaykkk4UicX2bLv1Qb1YF5eoWj/tE2 ZAa0ORNJ8b1GQpO/LOGZo3fTBHUPBsARKvjViKzZ+UquwosuFf9OQABeBIn/xOAyGicB vlvdDITzLP3subtegbjYr+e4wl91Pu/Tvd0cldkhG8jMzhMwh1uyg6Djq/YcNukH+boM UI9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721197568; x=1721802368; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3zMQJJ1+QlmFm418V6AGBGbWQY3rNoVY0h/UyfCjPDE=; b=q0L5HsuXZbCugxoV5BeBwkRPhj7x9JgwhC8TiyiEv23vQOVLEcdHNTahcSXlgIfYZx IttWR26CzuI+xdWr2K83MJQgdzSpzBCLUlQA+LxcZ3D0UGeTUXuFA0/UF1EMWGKq/M1k hbVBeg+/aVS/eCjzfb4f5pA+rQqkJO5sUBHfX6MftABi/Tba/H7HHBeXmZQ6Ku1+ffGc BBqCft3kFp0Dwc2xIg/G/of97bx1jhMcCuD83Aomrh5goWSPx6qxP6CxvTRAh0MIFioR QGo0peLY+rvTW1APV2RiUNw9o/k40EuC4GiHa6BqPwGQ1466CmtIcCJuqntZwL78HYI9 +bpQ== X-Forwarded-Encrypted: i=1; AJvYcCX0etaCO83mubZONJ4nnNj3D6UB+8nPzNULUixySgRrjnhwvDRyFmtOjfWBR1CVszVzdOeCInYZ5dbss6WvwGMlRydBsRVwyFvgsbJm X-Gm-Message-State: AOJu0Yz375VAVMg8cx5sENhg+JIkwkfQkpfKb/Zsopplpdfl2EJsHN4Q 2bxFXTI1HDi7ppiicNqPaKFYN9QYZ28is+B81IlllcPnjHKGKm9gdthfCJQD3uY= X-Google-Smtp-Source: AGHT+IE6uNjEd3rgzo2cnMZIenDe0Lf33upNb/sgH/fJlHpS9fKERnXLFFmQs0acmED+vAQAC04L2Q== X-Received: by 2002:adf:e681:0:b0:368:3079:4277 with SMTP id ffacd0b85a97d-3683165b50amr545623f8f.28.1721197567667; Tue, 16 Jul 2024 23:26:07 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-368111d3e7dsm10358022f8f.87.2024.07.16.23.26.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 23:26:07 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v3 06/11] riscv: Implement xchg8/16() using Zabha Date: Wed, 17 Jul 2024 08:19:52 +0200 Message-Id: <20240717061957.140712-7-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240717061957.140712-1-alexghiti@rivosinc.com> References: <20240717061957.140712-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This adds runtime support for Zabha in xchg8/16() operations. Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/cmpxchg.h | 32 +++++++++++++++++++++++++++++--- 1 file changed, 29 insertions(+), 3 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index 608d98522557..091e6612ddb3 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -11,8 +11,27 @@ #include #include =20 -#define __arch_xchg_masked(sc_sfx, prepend, append, r, p, n) \ +#define __arch_xchg_masked(sc_sfx, swap_sfx, prepend, sc_append, \ + swap_append, r, p, n) \ ({ \ + __label__ no_zabha, end; \ + \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA)) { \ + asm goto(ALTERNATIVE("j %[no_zabha]", "nop", 0, \ + RISCV_ISA_EXT_ZABHA, 1) \ + : : : : no_zabha); \ + \ + __asm__ __volatile__ ( \ + prepend \ + " amoswap" swap_sfx " %0, %z2, %1\n" \ + swap_append \ + : "=3D&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + goto end; \ + } \ + \ +no_zabha:; \ u32 *__ptr32b =3D (u32 *)((ulong)(p) & ~0x3); \ ulong __s =3D ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ ulong __mask =3D GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ @@ -28,12 +47,14 @@ " or %1, %1, %z3\n" \ " sc.w" sc_sfx " %1, %1, %2\n" \ " bnez %1, 0b\n" \ - append \ + sc_append \ : "=3D&r" (__retx), "=3D&r" (__rc), "+A" (*(__ptr32b)) \ : "rJ" (__newx), "rJ" (~__mask) \ : "memory"); \ \ r =3D (__typeof__(*(p)))((__retx & __mask) >> __s); \ + \ +end:; \ }) =20 #define __arch_xchg(sfx, prepend, append, r, p, n) \ @@ -56,8 +77,13 @@ \ switch (sizeof(*__ptr)) { \ case 1: \ + __arch_xchg_masked(sc_sfx, ".b" swap_sfx, \ + prepend, sc_append, swap_append, \ + __ret, __ptr, __new); \ + break; \ case 2: \ - __arch_xchg_masked(sc_sfx, prepend, sc_append, \ + __arch_xchg_masked(sc_sfx, ".h" swap_sfx, \ + prepend, sc_append, swap_append, \ __ret, __ptr, __new); \ break; \ case 4: \ --=20 2.39.2 From nobody Thu Dec 18 08:34:04 2025 Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8D09E171C2 for ; Wed, 17 Jul 2024 06:27:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721197632; cv=none; b=Kn4fYjvjBfCkkB3KB1oIG3BK48ZZz5zLXNPOaXuV+nn64hL+sVbWCqJrFmFLBLj9wJtOXZSBoSrS49C+o8dalszfckm4V/cu5pohucAeRrSOUYHPKLQO7ukC79WWwGd4swDBnf2R7goxXKwGBhKK9oEr3xuYwaxZ9cqMYy/GOIU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721197632; c=relaxed/simple; bh=zIctHSF8s4ycH2M1Xuee5DajFIDyCPEPGL7Vxs9tOR4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=DYCYFsO1U8H91UiUm9mRmLRYCMMzQTYoWl40E/lrz0POzLegz43EUtSdoJTsPfLnX+SMFenhnpAuSFbYOj3lyUPN6mqvxC99/RNSngLLT2ZgvUGhdqjd1A1s7aHPlYUQ9TYdCMN+Cyaw6UYZqv8sAB6/qYKR8EuDpYqaU3T/jO0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=oluJwtGz; arc=none smtp.client-ip=209.85.221.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="oluJwtGz" Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-368380828d6so63069f8f.1 for ; Tue, 16 Jul 2024 23:27:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1721197629; x=1721802429; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=heAbIXv1oMx4SoDoWG1mJ8ALiEwmwn+mI3czPDKQjFY=; b=oluJwtGzW8UFQJOGpztldH/7EktLklT6yBARhB88b39iASMTLXUasaGTQZ8rJMVLEN OAqRSJIRhoGDeUMECAe6gMReMMMxHhN8l2DzHVVxWj6V6E+FYriT+oEueSPu6vRAGqVO KIK+L0XVbxTfVEHKG3ci3u9Xo7Oper/bbVIij2l3x8HPYzkUzbmEjLZXxy59aHqrvlom r6WqnF9+JYzisGwX5ekGYLRfl1kaJvx7sLOGszf8WXGQtiOiuIJTS2VnpPcQDdA6cUQC GuPAraiuz16MQiG68cqh8J40ag3X69ekVeHJ09c7HmcjuRPvquhcTzHo4PIWmUpi58zR 6guQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721197629; x=1721802429; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=heAbIXv1oMx4SoDoWG1mJ8ALiEwmwn+mI3czPDKQjFY=; b=oYZZxmWbZvyZICE7yQQr0x4DTbmhrsAEUwh4/N5/ZcLhhWaO8DU2SQZDb29EHrTV4l b8tyK9uIdWhlnEQxSfev+i944Mgfo3zBDCGEnpNt4u5ZxWjvaTvvK6MHnPPUKeUwn1Pn kRxbKjc02YmptymnO76+QGB9JoInAb/0eTzFB/fRoRrJGlJhUyfC2igtNIf0zgnf1hnt /RYJzjxBTY05q+rdJYbeoKE3UK0llit5FQJuAx8k5dLJwFud4RVjOqKTIf3ca6oj9mCQ CwR+LvxMKYvtP1h1oLXWzFArLuI77rrrOacpK5p5lvxi38jg7vo/AOOVo1sn0VC254JQ QC4g== X-Forwarded-Encrypted: i=1; AJvYcCVCzkpsAVofXPgVxA2pkbvW1Os4yG6svTIpODvZr08qaxZWRe+u2wZ5Uvi3L4T+reDKzR8GXEizxwbsBVWaio/ZzEiPIS2pvcsiRing X-Gm-Message-State: AOJu0YxFdLTuQ1Aj/Nr8UKDOlBMRCB805Y29ikFPn/vrNqdGQV10A6ik slRCHX0y8/w8DuTRTk1oySDgYjVQtTTe4GIeeHlMNqTzLEn9k/GlyaTqfb3y9XI= X-Google-Smtp-Source: AGHT+IEcx5vbXw9AngQayUYduH0av6OpvecXDO/fzCki4vNwqd2UbQND86b7G8WFV0E/r2hUOY9S4Q== X-Received: by 2002:adf:f6c8:0:b0:367:bb20:b3e1 with SMTP id ffacd0b85a97d-3683171f8bemr430134f8f.51.1721197628922; Tue, 16 Jul 2024 23:27:08 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-427a5e8e2ecsm160904995e9.21.2024.07.16.23.27.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 23:27:08 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Guo Ren Subject: [PATCH v3 07/11] asm-generic: ticket-lock: Reuse arch_spinlock_t of qspinlock Date: Wed, 17 Jul 2024 08:19:53 +0200 Message-Id: <20240717061957.140712-8-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240717061957.140712-1-alexghiti@rivosinc.com> References: <20240717061957.140712-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Guo Ren The arch_spinlock_t of qspinlock has contained the atomic_t val, which satisfies the ticket-lock requirement. Thus, unify the arch_spinlock_t into qspinlock_types.h. This is the preparation for the next combo spinlock. Reviewed-by: Leonardo Bras Suggested-by: Arnd Bergmann Link: https://lore.kernel.org/linux-riscv/CAK8P3a2rnz9mQqhN6-e0CGUUv9rntREL= Fdxt_weiD7FxH7fkfQ@mail.gmail.com/ Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- include/asm-generic/spinlock.h | 14 +++++++------- include/asm-generic/spinlock_types.h | 12 ++---------- 2 files changed, 9 insertions(+), 17 deletions(-) diff --git a/include/asm-generic/spinlock.h b/include/asm-generic/spinlock.h index 90803a826ba0..4773334ee638 100644 --- a/include/asm-generic/spinlock.h +++ b/include/asm-generic/spinlock.h @@ -32,7 +32,7 @@ =20 static __always_inline void arch_spin_lock(arch_spinlock_t *lock) { - u32 val =3D atomic_fetch_add(1<<16, lock); + u32 val =3D atomic_fetch_add(1<<16, &lock->val); u16 ticket =3D val >> 16; =20 if (ticket =3D=3D (u16)val) @@ -46,31 +46,31 @@ static __always_inline void arch_spin_lock(arch_spinloc= k_t *lock) * have no outstanding writes due to the atomic_fetch_add() the extra * orderings are free. */ - atomic_cond_read_acquire(lock, ticket =3D=3D (u16)VAL); + atomic_cond_read_acquire(&lock->val, ticket =3D=3D (u16)VAL); smp_mb(); } =20 static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) { - u32 old =3D atomic_read(lock); + u32 old =3D atomic_read(&lock->val); =20 if ((old >> 16) !=3D (old & 0xffff)) return false; =20 - return atomic_try_cmpxchg(lock, &old, old + (1<<16)); /* SC, for RCsc */ + return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RC= sc */ } =20 static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) { u16 *ptr =3D (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); - u32 val =3D atomic_read(lock); + u32 val =3D atomic_read(&lock->val); =20 smp_store_release(ptr, (u16)val + 1); } =20 static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) { - u32 val =3D lock.counter; + u32 val =3D lock.val.counter; =20 return ((val >> 16) =3D=3D (val & 0xffff)); } @@ -84,7 +84,7 @@ static __always_inline int arch_spin_is_locked(arch_spinl= ock_t *lock) =20 static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) { - u32 val =3D atomic_read(lock); + u32 val =3D atomic_read(&lock->val); =20 return (s16)((val >> 16) - (val & 0xffff)) > 1; } diff --git a/include/asm-generic/spinlock_types.h b/include/asm-generic/spi= nlock_types.h index 8962bb730945..f534aa5de394 100644 --- a/include/asm-generic/spinlock_types.h +++ b/include/asm-generic/spinlock_types.h @@ -3,15 +3,7 @@ #ifndef __ASM_GENERIC_SPINLOCK_TYPES_H #define __ASM_GENERIC_SPINLOCK_TYPES_H =20 -#include -typedef atomic_t arch_spinlock_t; - -/* - * qrwlock_types depends on arch_spinlock_t, so we must typedef that befor= e the - * include. - */ -#include - -#define __ARCH_SPIN_LOCK_UNLOCKED ATOMIC_INIT(0) +#include +#include =20 #endif /* __ASM_GENERIC_SPINLOCK_TYPES_H */ --=20 2.39.2 From nobody Thu Dec 18 08:34:04 2025 Received: from mail-wr1-f42.google.com (mail-wr1-f42.google.com [209.85.221.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E1E2612B6C for ; Wed, 17 Jul 2024 06:28:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721197693; cv=none; b=VBcCeX3e+vriY/f+U2v10zVifVbPSKXMrADrrfNJOeucC32+YsNzAVeEqKkceSI+zqmQ7U6k3G0/+JUM4iO5rFdDw+SMCRM7FK8e6JVlMaK+6tWYR+P4wYUoj0a2fpYEfNY8dvY3vNZR/ldl1ko2Y3g2WwSswo2Z7CT2Ztv+IqA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721197693; c=relaxed/simple; bh=o4ZK/GG6lXAN1cfDXgIpVYZWTFwSck3jVd1HEtW4i7A=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=bhb0cWBQgBdVEyPMKmgWh3S9Ldj1iTHXpQ0uepIoGSXi2teYd2IhmT6cyxkRYbfB+sg2rJaNhrXxM+H1j2RjSvN99sU3fsibhOGPXMryhdKbcvPSgCDp4KOmYjA5jWBWJX/bGJEdC0LOzXwLO2lD5flV4y8Ot5G6EUKuCplSlvg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=I6u/wuUj; arc=none smtp.client-ip=209.85.221.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="I6u/wuUj" Received: by mail-wr1-f42.google.com with SMTP id ffacd0b85a97d-36796aee597so3855648f8f.1 for ; Tue, 16 Jul 2024 23:28:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1721197690; x=1721802490; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JpMFH6VmSNUk91jIQ+0kjGx11FiPCz2ltKgHshZWUPg=; b=I6u/wuUj5CnDUwLH4wrq9HfUioSQ7oasAohffOGS68gfC9zsa5layZnUFJOwA/w0rM Tt5VSRxm+G5nL4CjwLnKaB53jOkz/niuppF0xoU77TOuVfXxwkbMIuWiQZQP/G/dE7hd uLBaJOoOEb1faae/bPdA/QDGjITikMzGsJC5Fqz9hB5hH6T5ZCoq/DM14ohxn2YYDS2H /6FVwKbNu59UGmT56iK12KaxIO8p8QvtDeMTaDyOak/OJrRlxnCAp4g/6BBqJzGNZvkp jpi3DobF56AD3OXStSvtr7RDbRMMHEwk8v0WYoQlJI3+dk399zAiq2KO3dJC81AePHUW KT5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721197690; x=1721802490; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JpMFH6VmSNUk91jIQ+0kjGx11FiPCz2ltKgHshZWUPg=; b=Qm6/Bkmm124FFgB3JBxtd59SkVSRAaVVNZXzP5GxmXR4YitqJN+BWtpB7uSMgT0H8r LyxnlRYYYW+a88DVD7RTqMldwNrSVF3LI8urBbxUZx0i+PdkhdMUcmkf5VfpgGyIqPFq b5l8joBYqg2sEV9b+3pb7GLuOx+Wz48tK0DzYvRMAUU70fmPbC0Zh4hv6bFGgEvAfvKX wAiRYRAL57/SabW2uciNlZ39V/wDWT3imY8OqcQocXyeACpS+VCn+QZggUsdAriolZVV NGjZjD5kzMU/jOwFMLi5jeQ0T7PxCbmd/eERbgqUyTAZ5q+JSjiCeiq3Pbnu0DnkQAOZ 4krQ== X-Forwarded-Encrypted: i=1; AJvYcCWPrYvlFV/Nef6W8MGHScpAx0eUJ7VXFePIvKzXU3gNgi1+NWttykdiXFGdtyicvPZ1nb+ycdgxR7LOPlYTp3p48evl/hhTHJFULkWJ X-Gm-Message-State: AOJu0YypnbclBptPBlatuhQHR9eeVijqKt5eaYMmbHY2ZSoUSqbSHOuY yqdKqB+/o2ChB3P15m0L1Qrq+YOPUJzQR+zCqIATJz1wNplQHPWz6OGuLDvQJ9s= X-Google-Smtp-Source: AGHT+IFM/AGZ5F1dS74/5pYkBU8cAcbJDWrIIu9c3nYaG0zPFdfcFYCGZfuTjOfFxaw1xsX/SOzMGA== X-Received: by 2002:adf:e508:0:b0:366:f041:935d with SMTP id ffacd0b85a97d-3683179ebc7mr379203f8f.60.1721197690168; Tue, 16 Jul 2024 23:28:10 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3680daccacdsm10824862f8f.62.2024.07.16.23.28.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 23:28:09 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Guo Ren Subject: [PATCH v3 08/11] asm-generic: ticket-lock: Add separate ticket-lock.h Date: Wed, 17 Jul 2024 08:19:54 +0200 Message-Id: <20240717061957.140712-9-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240717061957.140712-1-alexghiti@rivosinc.com> References: <20240717061957.140712-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Guo Ren Add a separate ticket-lock.h to include multiple spinlock versions and select one at compile time or runtime. Reviewed-by: Leonardo Bras Suggested-by: Arnd Bergmann Link: https://lore.kernel.org/linux-riscv/CAK8P3a2rnz9mQqhN6-e0CGUUv9rntREL= Fdxt_weiD7FxH7fkfQ@mail.gmail.com/ Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- include/asm-generic/spinlock.h | 87 +--------------------- include/asm-generic/ticket_spinlock.h | 103 ++++++++++++++++++++++++++ 2 files changed, 104 insertions(+), 86 deletions(-) create mode 100644 include/asm-generic/ticket_spinlock.h diff --git a/include/asm-generic/spinlock.h b/include/asm-generic/spinlock.h index 4773334ee638..970590baf61b 100644 --- a/include/asm-generic/spinlock.h +++ b/include/asm-generic/spinlock.h @@ -1,94 +1,9 @@ /* SPDX-License-Identifier: GPL-2.0 */ =20 -/* - * 'Generic' ticket-lock implementation. - * - * It relies on atomic_fetch_add() having well defined forward progress - * guarantees under contention. If your architecture cannot provide this, = stick - * to a test-and-set lock. - * - * It also relies on atomic_fetch_add() being safe vs smp_store_release() = on a - * sub-word of the value. This is generally true for anything LL/SC althou= gh - * you'd be hard pressed to find anything useful in architecture specifica= tions - * about this. If your architecture cannot do this you might be better off= with - * a test-and-set. - * - * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and = hence - * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along= with - * a full fence after the spin to upgrade the otherwise-RCpc - * atomic_cond_read_acquire(). - * - * The implementation uses smp_cond_load_acquire() to spin, so if the - * architecture has WFE like instructions to sleep instead of poll for word - * modifications be sure to implement that (see ARM64 for example). - * - */ - #ifndef __ASM_GENERIC_SPINLOCK_H #define __ASM_GENERIC_SPINLOCK_H =20 -#include -#include - -static __always_inline void arch_spin_lock(arch_spinlock_t *lock) -{ - u32 val =3D atomic_fetch_add(1<<16, &lock->val); - u16 ticket =3D val >> 16; - - if (ticket =3D=3D (u16)val) - return; - - /* - * atomic_cond_read_acquire() is RCpc, but rather than defining a - * custom cond_read_rcsc() here we just emit a full fence. We only - * need the prior reads before subsequent writes ordering from - * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we - * have no outstanding writes due to the atomic_fetch_add() the extra - * orderings are free. - */ - atomic_cond_read_acquire(&lock->val, ticket =3D=3D (u16)VAL); - smp_mb(); -} - -static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) -{ - u32 old =3D atomic_read(&lock->val); - - if ((old >> 16) !=3D (old & 0xffff)) - return false; - - return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RC= sc */ -} - -static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) -{ - u16 *ptr =3D (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); - u32 val =3D atomic_read(&lock->val); - - smp_store_release(ptr, (u16)val + 1); -} - -static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) -{ - u32 val =3D lock.val.counter; - - return ((val >> 16) =3D=3D (val & 0xffff)); -} - -static __always_inline int arch_spin_is_locked(arch_spinlock_t *lock) -{ - arch_spinlock_t val =3D READ_ONCE(*lock); - - return !arch_spin_value_unlocked(val); -} - -static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) -{ - u32 val =3D atomic_read(&lock->val); - - return (s16)((val >> 16) - (val & 0xffff)) > 1; -} - +#include #include =20 #endif /* __ASM_GENERIC_SPINLOCK_H */ diff --git a/include/asm-generic/ticket_spinlock.h b/include/asm-generic/ti= cket_spinlock.h new file mode 100644 index 000000000000..cfcff22b37b3 --- /dev/null +++ b/include/asm-generic/ticket_spinlock.h @@ -0,0 +1,103 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * 'Generic' ticket-lock implementation. + * + * It relies on atomic_fetch_add() having well defined forward progress + * guarantees under contention. If your architecture cannot provide this, = stick + * to a test-and-set lock. + * + * It also relies on atomic_fetch_add() being safe vs smp_store_release() = on a + * sub-word of the value. This is generally true for anything LL/SC althou= gh + * you'd be hard pressed to find anything useful in architecture specifica= tions + * about this. If your architecture cannot do this you might be better off= with + * a test-and-set. + * + * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and = hence + * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along= with + * a full fence after the spin to upgrade the otherwise-RCpc + * atomic_cond_read_acquire(). + * + * The implementation uses smp_cond_load_acquire() to spin, so if the + * architecture has WFE like instructions to sleep instead of poll for word + * modifications be sure to implement that (see ARM64 for example). + * + */ + +#ifndef __ASM_GENERIC_TICKET_SPINLOCK_H +#define __ASM_GENERIC_TICKET_SPINLOCK_H + +#include +#include + +static __always_inline void ticket_spin_lock(arch_spinlock_t *lock) +{ + u32 val =3D atomic_fetch_add(1<<16, &lock->val); + u16 ticket =3D val >> 16; + + if (ticket =3D=3D (u16)val) + return; + + /* + * atomic_cond_read_acquire() is RCpc, but rather than defining a + * custom cond_read_rcsc() here we just emit a full fence. We only + * need the prior reads before subsequent writes ordering from + * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we + * have no outstanding writes due to the atomic_fetch_add() the extra + * orderings are free. + */ + atomic_cond_read_acquire(&lock->val, ticket =3D=3D (u16)VAL); + smp_mb(); +} + +static __always_inline bool ticket_spin_trylock(arch_spinlock_t *lock) +{ + u32 old =3D atomic_read(&lock->val); + + if ((old >> 16) !=3D (old & 0xffff)) + return false; + + return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RC= sc */ +} + +static __always_inline void ticket_spin_unlock(arch_spinlock_t *lock) +{ + u16 *ptr =3D (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); + u32 val =3D atomic_read(&lock->val); + + smp_store_release(ptr, (u16)val + 1); +} + +static __always_inline int ticket_spin_value_unlocked(arch_spinlock_t lock) +{ + u32 val =3D lock.val.counter; + + return ((val >> 16) =3D=3D (val & 0xffff)); +} + +static __always_inline int ticket_spin_is_locked(arch_spinlock_t *lock) +{ + arch_spinlock_t val =3D READ_ONCE(*lock); + + return !ticket_spin_value_unlocked(val); +} + +static __always_inline int ticket_spin_is_contended(arch_spinlock_t *lock) +{ + u32 val =3D atomic_read(&lock->val); + + return (s16)((val >> 16) - (val & 0xffff)) > 1; +} + +/* + * Remapping spinlock architecture specific functions to the corresponding + * ticket spinlock functions. + */ +#define arch_spin_is_locked(l) ticket_spin_is_locked(l) +#define arch_spin_is_contended(l) ticket_spin_is_contended(l) +#define arch_spin_value_unlocked(l) ticket_spin_value_unlocked(l) +#define arch_spin_lock(l) ticket_spin_lock(l) +#define arch_spin_trylock(l) ticket_spin_trylock(l) +#define arch_spin_unlock(l) ticket_spin_unlock(l) + +#endif /* __ASM_GENERIC_TICKET_SPINLOCK_H */ --=20 2.39.2 From nobody Thu Dec 18 08:34:04 2025 Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2448C182B9 for ; Wed, 17 Jul 2024 06:29:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721197754; cv=none; b=uaVBRSpRhBQHXVKkNCxmeSTunnJHq2MUTGVozeN7PJjB5BosGf9K9YgbdkoZaM8tmw1IcFc1UflH1p5DoqbtlZsmIbj/etVXObqmIeaStw+zhk3jQ/8TJgFqxjjCnLnMA7E7YogXMWH/T6HmVGXW/1kaz/bEH7p0x7X8OULxHb0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721197754; c=relaxed/simple; bh=piibWXQoIdT9iDF1Ucz8hY3qyEnh4/nv/5EUjy5T18E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=RTldHQIhLUpremATuehUOV2lPvSNShm41IpcuvtlirOL2QjQdoM7V4spJMkvtm0UKDQKlrcqnTqR0S/U5SCHfrgSMwZednN9KJmHIc67KRAjNBGn6WILhAdbXTPSSDQe1SA5uIGGkKUnSuAUq0t8F23NGsDgY42199r3sMmVdC4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=cFo/1ZBX; arc=none smtp.client-ip=209.85.221.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="cFo/1ZBX" Received: by mail-wr1-f52.google.com with SMTP id ffacd0b85a97d-367a183df54so4253249f8f.2 for ; Tue, 16 Jul 2024 23:29:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1721197751; x=1721802551; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qV10XIriQxrkSLT6hfJ9bzuJQGW5jGWUcHq8Bf3DvNw=; b=cFo/1ZBXvHn/hpmxtuTuvd40MaVXd2ic+M8iCgODTXyAbHYx3UBS3adMgozTX13xU9 icO4TXiea9Ox/i/dSj9GpHP/KmiRygne4Nf9cjw2yqOoyijDLEFT/Wx9WTUVjKn7uRdo MArV0oD7iufkh+YiR7p7FogMtsi68d/FiLce3aGS3KdVQH+ng5wf4yfrOL/DlFvJZ0Mh vNXzOlkk5sHbX4Mli6Y29oaBMoNfHcJaDoa0zTsa00VCysbAJkpBrJF2azpTzyTLbduW aq8z6jjFK2Cb250gntgt/22ApB7soebI4C8jyVAsR88pJUFCGZ/RRUbqqUsra3WRxlcJ m+PA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721197751; x=1721802551; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qV10XIriQxrkSLT6hfJ9bzuJQGW5jGWUcHq8Bf3DvNw=; b=oamqx1ZP1qzBeorzf5GDL8YeJ05jYcNvqlDxDplxc2mZ8ENm7EqnTla2N6om8USFbU d/qur6wDCyV0hm3Tncb+XwOLJrAbeX8ZWpMDRvw7LhuZ9IM+stP5zndwUx/59KvzY57w ZaXT8cpO4xe3RAPGryWr172dgTV7/3cDJ6mTDUeMtz4KWe/thl2/hG19ai7eX9yTlmiV 0x8+zY8690G80KWoh7nO6ByjdycGnObM5kHFR+iTu0LgFZYgNsE92+QlwyfbvW/1Wp6e cAcPV6akKsARxBIA8/uUpxNPVABfO5ApMMB3Zge2eKs494pYu/Xh9v6/1kWvOh5b1TWc uA8g== X-Forwarded-Encrypted: i=1; AJvYcCWatirVK4j/rkfe+qVWM745nvB3QtDtUW7bMXdu9GBXBwZ0b4JjKUcAVExRaL9HmgG6g37PzmUJaHGQADZQ8+MT4W/uOvTPKBs/gDxr X-Gm-Message-State: AOJu0Yw1uPmg8e4FykkTFAM+7lygEUJJ+s9DMUIWtuP700Crf+t59a9y S2UFUb3nTUeHNJErd/UGoHZkwzo5bUC2zxm8SIPg1ESGR0ffSVtQunjWx+rTyFY= X-Google-Smtp-Source: AGHT+IGP/cuCt+F0ta7uxUlFKhwXsnKpljMYZclyaPYQLOpIG5TT5vjECTCQ1Rcr5BTRTziHhuytmQ== X-Received: by 2002:adf:fa03:0:b0:368:303b:8fe7 with SMTP id ffacd0b85a97d-36831600243mr520432f8f.7.1721197751437; Tue, 16 Jul 2024 23:29:11 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3680daccad8sm10840137f8f.60.2024.07.16.23.29.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 23:29:11 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v3 09/11] riscv: Add ISA extension parsing for Ziccrse Date: Wed, 17 Jul 2024 08:19:55 +0200 Message-Id: <20240717061957.140712-10-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240717061957.140712-1-alexghiti@rivosinc.com> References: <20240717061957.140712-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add support to parse the Ziccrse string in the riscv,isa string. Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/kernel/cpufeature.c | 1 + 2 files changed, 2 insertions(+) diff --git a/arch/riscv/include/asm/hwcap.h b/arch/riscv/include/asm/hwcap.h index f71ddd2ca163..863b9b7d4a4f 100644 --- a/arch/riscv/include/asm/hwcap.h +++ b/arch/riscv/include/asm/hwcap.h @@ -82,6 +82,7 @@ #define RISCV_ISA_EXT_ZACAS 73 #define RISCV_ISA_EXT_XANDESPMU 74 #define RISCV_ISA_EXT_ZABHA 75 +#define RISCV_ISA_EXT_ZICCRSE 76 =20 #define RISCV_ISA_EXT_XLINUXENVCFG 127 =20 diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index c125d82c894b..93d8cc7e232c 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -306,6 +306,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] =3D { __RISCV_ISA_EXT_DATA(svnapot, RISCV_ISA_EXT_SVNAPOT), __RISCV_ISA_EXT_DATA(svpbmt, RISCV_ISA_EXT_SVPBMT), __RISCV_ISA_EXT_DATA(xandespmu, RISCV_ISA_EXT_XANDESPMU), + __RISCV_ISA_EXT_DATA(ziccrse, RISCV_ISA_EXT_ZICCRSE), }; =20 const size_t riscv_isa_ext_count =3D ARRAY_SIZE(riscv_isa_ext); --=20 2.39.2 From nobody Thu Dec 18 08:34:04 2025 Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 42AEA2FB6 for ; Wed, 17 Jul 2024 06:30:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721197816; cv=none; b=cGU6N0LlxOl4E4rPFAls2SO5/CzIMB13ik4wglWz6b8zUadjE/45cyhGbwOuY6VSXUhQb8m+fEXjts15lpf5iun3x2sGVG5nvp4/r+YV093uqTy63iS9PrGfv3shMuMxApRW68dyCWeUrr4dR+jhYNUE7SmBJ7UHCeXJMci7cM0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721197816; c=relaxed/simple; bh=V9Jzk4N2zS36v1WQ5gEhoUIZc3Ly89MBgolvymqDsAA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=YJMJeBzQFi2XWmeop14p5PplHKYqEAqQ9pxMiy4UxisKCZSzmHMlzblOaT7xb1KZeJxEK64l4LOr73QstEvQ/9CwemPAuvykTLWuRvFsEm7/2PlRMBC3E1SOY1nEK+VBreCxE8J+CM9wfiKBjqpsYQ07os6Y/1TSNuEs+cjhsU0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=T4TxoWBi; arc=none smtp.client-ip=209.85.128.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="T4TxoWBi" Received: by mail-wm1-f44.google.com with SMTP id 5b1f17b1804b1-4266eda81c5so51437675e9.0 for ; Tue, 16 Jul 2024 23:30:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1721197813; x=1721802613; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=r/a3I9RG+3lbcPTW2WISOokc8gewDy3fkMsPDZhy/D4=; b=T4TxoWBiQ7J7z3doxhHqMilZccLpeo/vCwAtUpKuuP+myV49KcZikkAfThhiqEsZgU 6gfHD+PDyo1GRWtzVi/bDJjcfvGSPrsdcaNYgZI7URcw+9eBmXWcuQ0q7om8jO9IHNwR /AhsNZ3pwphzKrvWMD0YA/v4i+YuUQJNlbdnXg/xeIJnri1LCj0e+70b0JefrLhz+hkF F3yhc9wWcpbB1VcL9I8XLKJLCnDzqEwaVibgXBQjMTGLbgPbgT7/8dc2vKwGVlBAAnAm B1myDcVgh3Cn38o4vXgcMZgWGIVTjKNizRWXJu5cvIUGpSnv7OKDdupQ0sAJTGahp3ZO rdWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721197813; x=1721802613; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=r/a3I9RG+3lbcPTW2WISOokc8gewDy3fkMsPDZhy/D4=; b=soVj9x1Xfdsyo4nQQa2Ga6BAcB6CPetZF+Wtuto58XIxxLsi1CO+D7+xXfGBWo4zQM HUHfTd3CLTPzaBdPTUSkwXYCEI+gT6vNo9Lvahf8aOSUIXl67eyZbjE6wqsJ9UAssEpx TgPHtv8Q2ZurHym1aXJHEqJN0k890eDJIx3S7g4pZXrK1tZZbPD0ESvNW1kSAYyNtexn RqOHBaZBj/UKTxYp3GNSLjZDM5ZuEpNLC5z8scXm4sj8b+EpsVWTlAlpQiCuUVUSKdBm e8uI0bwgMAoCbKbXWapTNJjzpwSI/HPGyAYXiG1WLQgwPpur4B7g7AlUr86anYsaaf0I KJkQ== X-Forwarded-Encrypted: i=1; AJvYcCXY5VA9wUa9wyriw7QnDsAVmEAVd9PmY1TB+sDhidDzQdIjKt6NOONJf+JnuYEbIPXLDlakDdrjJmXRa5xNEp77VpEZZGnVJnuUlMVO X-Gm-Message-State: AOJu0Yx1mtdXd9C9BssORVmE7bZ5D//k0oSsQ2ndC93UtutqatJ6GiNo B71oZEhPkeLQDYMQIVfvRUr5i2KW1TgRdfjpGlTz4o5qiTjCKf+R9J1tDab5Azo= X-Google-Smtp-Source: AGHT+IHes7c6JkEf7yEOk72RnIcqQqTVYNuy3gJbWA9je9hHeJGPheqNOo3gwFeqz1xI+QDtIGxtPA== X-Received: by 2002:a05:600c:4ec9:b0:426:62c5:4731 with SMTP id 5b1f17b1804b1-427c2d01cafmr5720315e9.29.1721197812676; Tue, 16 Jul 2024 23:30:12 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-427a5ef460csm158127685e9.44.2024.07.16.23.30.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 23:30:12 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v3 10/11] dt-bindings: riscv: Add Ziccrse ISA extension description Date: Wed, 17 Jul 2024 08:19:56 +0200 Message-Id: <20240717061957.140712-11-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240717061957.140712-1-alexghiti@rivosinc.com> References: <20240717061957.140712-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add description for the Ziccrse ISA extension which was introduced in the riscv profiles specification v0.9.2. Signed-off-by: Alexandre Ghiti Reviewed-by: Guo Ren --- Documentation/devicetree/bindings/riscv/extensions.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Documentation/devicetree/bindings/riscv/extensions.yaml b/Docu= mentation/devicetree/bindings/riscv/extensions.yaml index e6436260bdeb..b08bf1a8d8f8 100644 --- a/Documentation/devicetree/bindings/riscv/extensions.yaml +++ b/Documentation/devicetree/bindings/riscv/extensions.yaml @@ -245,6 +245,12 @@ properties: in commit 64074bc ("Update version numbers for Zfh/Zfinx") of riscv-isa-manual. =20 + - const: ziccrse + description: + The standard Ziccrse extension which provides forward progress + guarantee on LR/SC sequences, as introduced in the riscv profi= les + specification v0.9.2. + - const: zk description: The standard Zk Standard Scalar cryptography extension as rati= fied --=20 2.39.2 From nobody Thu Dec 18 08:34:04 2025 Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C5D7713ACC for ; Wed, 17 Jul 2024 06:31:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721197878; cv=none; b=UF0dTcU7ALWpoBUmtGSU76AMijGLUoE6/NBjfEgbugRkjSx+zuIptFYh0nvaeI3lNe0gJCdPnzxj5NZx875buk+8DS+ZhfaxOC1ITvTAC22uKQ551p8W9uQooseY5VUjq0Z64N0oTFP47EKdsYysL+/wHfPY+XplgQ3aDiCOUvw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721197878; c=relaxed/simple; bh=iw5RSspVlnVonfCFaM95ipa0asG/GpUdwDJ+E1tB1pc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rSd2s6Vua1+Ne8tWzAmJkArI9ei2rO8lWk7TT1NioUrln8w5TReZsavueMv3idxiVN4XpzUTNi4Y62JoNNxXoKU70Gt9xvqW2L+xm9KMnu/5YpcnBNjsI7eYDmXb/RXYKJV1J65GwrK345FSPX0t7BcSr0M6+DAMiYJYncdgjg4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=iUeSiMJo; arc=none smtp.client-ip=209.85.128.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="iUeSiMJo" Received: by mail-wm1-f54.google.com with SMTP id 5b1f17b1804b1-4267300145eso51285805e9.3 for ; Tue, 16 Jul 2024 23:31:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1721197874; x=1721802674; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kdclfDAD0BbbW9A8JO1TG59iSyD1aGCH+ewt/4gE4jc=; b=iUeSiMJoqqY4fbeRLkRmlyNrPsKq/MhxRQVmdyCZDf6KYtwYgJMosnNVP3fiL6lpBB 0EzgTYBrESPuc7FRDWsvvyrxGRkV7d78QqKCLLIa7czpG7cHZJNwypvMVNxTvFuMI22l Flc0AToMF4NH32AdIVwUDYY1qr0+rnLSjtz7ScTrpQr8As8rwdEEndcS398qQeWxpB/e 1InMo+irDI85LOyvG/loYerC968kDZfwXOFB3iCg8086sy1DCGZxkMEY3FKkR9Ekba3v JB+wBp9ks3Xvq5FoBX6Lx/N9AcAC5iic5TDONBudwrbe908VOQyvdOMSz1dtoeS5QnLh N1xA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721197874; x=1721802674; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kdclfDAD0BbbW9A8JO1TG59iSyD1aGCH+ewt/4gE4jc=; b=uhkptFBX+QlKJ77DqGyLTrBdarLori1WNdbxTbNV+mcTnDjpHIZ+jnKahLO2w1JzYY a8/+9xtzUM2MfyY8XSH19vfzo76CXCpk4ilv8WmJlOxsSc5ksNHrqQax8SMySktIW9Fp OL/zRAiQGo2RKzvVlTNHcoTDam0ww3qoKl+JidcOE97fkXbfvkVgOAxUar1VPsl9S68U H4paHNSsWuaBS9uahMhVBn66Fy/75mt5xb6bA09ZFG6gpHhfBr9mlxFMfQf7chaH41+3 66JeofyQQNj984WS8mp/l8sjJV3UPjaGnIcSFK2yqolBAyCxtODuSIU5o4PJIfZYGTDy aXlw== X-Forwarded-Encrypted: i=1; AJvYcCVkYELFJxBaiSDxA/+0d1fw+/8yAhktujm92Kf5ZzWwlkokm3oFFUTMDl8zjcC8n77Lfcq9q0TyRpfiw+OKMdBCKA50bbCcdYsnN5Jt X-Gm-Message-State: AOJu0YwGVTzRmBTypCPKYn+kWFDDugQhzvC1TNMr337vxcEVJjbl/NeQ zY6oO2ijO2u7g4U/+IhPrS0pCOgJEfF7iz9w1wn1a+OzlNhJJS6eMNu2089KM40= X-Google-Smtp-Source: AGHT+IEb50phVixq0u+U2husY+tMg8VyZk4HDBQadM7JWcW6QpG95LwceBZFv+wDTFAxqwuJF0ajmQ== X-Received: by 2002:a05:600c:3544:b0:426:6822:5aa8 with SMTP id 5b1f17b1804b1-427c2cc8481mr4819935e9.18.1721197874118; Tue, 16 Jul 2024 23:31:14 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-427a5e996c1sm159710625e9.25.2024.07.16.23.31.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 23:31:13 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v3 11/11] riscv: Add qspinlock support Date: Wed, 17 Jul 2024 08:19:57 +0200 Message-Id: <20240717061957.140712-12-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240717061957.140712-1-alexghiti@rivosinc.com> References: <20240717061957.140712-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In order to produce a generic kernel, a user can select CONFIG_COMBO_SPINLOCKS which will fallback at runtime to the ticket spinlock implementation if Zabha or Ziccrse are not present. Note that we can't use alternatives here because the discovery of extensions is done too late and we need to start with the qspinlock implementation because the ticket spinlock implementation would pollute the spinlock value, so let's use static keys. This is largely based on Guo's work and Leonardo reviews at [1]. Link: https://lore.kernel.org/linux-riscv/20231225125847.2778638-1-guoren@k= ernel.org/ [1] Signed-off-by: Guo Ren Signed-off-by: Alexandre Ghiti --- .../locking/queued-spinlocks/arch-support.txt | 2 +- arch/riscv/Kconfig | 29 ++++++++++++++ arch/riscv/include/asm/Kbuild | 4 +- arch/riscv/include/asm/spinlock.h | 39 +++++++++++++++++++ arch/riscv/kernel/setup.c | 33 ++++++++++++++++ include/asm-generic/qspinlock.h | 2 + include/asm-generic/ticket_spinlock.h | 2 + 7 files changed, 109 insertions(+), 2 deletions(-) create mode 100644 arch/riscv/include/asm/spinlock.h diff --git a/Documentation/features/locking/queued-spinlocks/arch-support.t= xt b/Documentation/features/locking/queued-spinlocks/arch-support.txt index 22f2990392ff..cf26042480e2 100644 --- a/Documentation/features/locking/queued-spinlocks/arch-support.txt +++ b/Documentation/features/locking/queued-spinlocks/arch-support.txt @@ -20,7 +20,7 @@ | openrisc: | ok | | parisc: | TODO | | powerpc: | ok | - | riscv: | TODO | + | riscv: | ok | | s390: | TODO | | sh: | TODO | | sparc: | ok | diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 0bbaec0444d0..5040c7eac70d 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -72,6 +72,7 @@ config RISCV select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP select ARCH_WANTS_NO_INSTR select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE + select ARCH_WEAK_RELEASE_ACQUIRE if ARCH_USE_QUEUED_SPINLOCKS select BINFMT_FLAT_NO_DATA_START_OFFSET if !MMU select BUILDTIME_TABLE_SORT if MMU select CLINT_TIMER if RISCV_M_MODE @@ -482,6 +483,34 @@ config NODES_SHIFT Specify the maximum number of NUMA Nodes available on the target system. Increases memory reserved to accommodate various tables. =20 +choice + prompt "RISC-V spinlock type" + default RISCV_COMBO_SPINLOCKS + +config RISCV_TICKET_SPINLOCKS + bool "Using ticket spinlock" + +config RISCV_QUEUED_SPINLOCKS + bool "Using queued spinlock" + depends on SMP && MMU + select ARCH_USE_QUEUED_SPINLOCKS + help + The queued spinlock implementation requires the forward progress + guarantee of cmpxchg()/xchg() atomic operations: CAS with Zabha or + LR/SC with Ziccrse provide such guarantee. + + Select this if and only if Zabha or Ziccrse is available on your + platform. + +config RISCV_COMBO_SPINLOCKS + bool "Using combo spinlock" + depends on SMP && MMU + select ARCH_USE_QUEUED_SPINLOCKS + help + Embed both queued spinlock and ticket lock so that the spinlock + implementation can be chosen at runtime. +endchoice + config RISCV_ALTERNATIVE bool depends on !XIP_KERNEL diff --git a/arch/riscv/include/asm/Kbuild b/arch/riscv/include/asm/Kbuild index 504f8b7e72d4..ad72f2bd4cc9 100644 --- a/arch/riscv/include/asm/Kbuild +++ b/arch/riscv/include/asm/Kbuild @@ -2,10 +2,12 @@ generic-y +=3D early_ioremap.h generic-y +=3D flat.h generic-y +=3D kvm_para.h +generic-y +=3D mcs_spinlock.h generic-y +=3D parport.h -generic-y +=3D spinlock.h generic-y +=3D spinlock_types.h +generic-y +=3D ticket_spinlock.h generic-y +=3D qrwlock.h generic-y +=3D qrwlock_types.h +generic-y +=3D qspinlock.h generic-y +=3D user.h generic-y +=3D vmlinux.lds.h diff --git a/arch/riscv/include/asm/spinlock.h b/arch/riscv/include/asm/spi= nlock.h new file mode 100644 index 000000000000..4856d50006f2 --- /dev/null +++ b/arch/riscv/include/asm/spinlock.h @@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __ASM_RISCV_SPINLOCK_H +#define __ASM_RISCV_SPINLOCK_H + +#ifdef CONFIG_RISCV_COMBO_SPINLOCKS +#define _Q_PENDING_LOOPS (1 << 9) + +#define __no_arch_spinlock_redefine +#include +#include +#include + +DECLARE_STATIC_KEY_TRUE(qspinlock_key); + +#define SPINLOCK_BASE_DECLARE(op, type, type_lock) \ +static __always_inline type arch_spin_##op(type_lock lock) \ +{ \ + if (static_branch_unlikely(&qspinlock_key)) \ + return queued_spin_##op(lock); \ + return ticket_spin_##op(lock); \ +} + +SPINLOCK_BASE_DECLARE(lock, void, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(unlock, void, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(is_locked, int, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(is_contended, int, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(trylock, bool, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(value_unlocked, int, arch_spinlock_t) + +#else + +#include + +#endif + +#include + +#endif /* __ASM_RISCV_SPINLOCK_H */ diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c index 4f73c0ae44b2..d7c31c9b8ead 100644 --- a/arch/riscv/kernel/setup.c +++ b/arch/riscv/kernel/setup.c @@ -244,6 +244,38 @@ static void __init parse_dtb(void) #endif } =20 +DEFINE_STATIC_KEY_TRUE(qspinlock_key); +EXPORT_SYMBOL(qspinlock_key); + +static void __init riscv_spinlock_init(void) +{ + char *using_ext; + + if (IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && + IS_ENABLED(CONFIG_RISCV_ISA_ZABHA)) { + using_ext =3D "using Zabha"; + + asm goto(ALTERNATIVE("j %[no_zacas]", "nop", 0, RISCV_ISA_EXT_ZACAS, 1) + : : : : no_zacas); + asm goto(ALTERNATIVE("nop", "j %[qspinlock]", 0, RISCV_ISA_EXT_ZABHA, 1) + : : : : qspinlock); + } + +no_zacas: + using_ext =3D "using Ziccrse"; + asm goto(ALTERNATIVE("nop", "j %[qspinlock]", 0, + RISCV_ISA_EXT_ZICCRSE, 1) + : : : : qspinlock); + + static_branch_disable(&qspinlock_key); + pr_info("Ticket spinlock: enabled\n"); + + return; + +qspinlock: + pr_info("Queued spinlock %s: enabled\n", using_ext); +} + extern void __init init_rt_signal_env(void); =20 void __init setup_arch(char **cmdline_p) @@ -295,6 +327,7 @@ void __init setup_arch(char **cmdline_p) riscv_set_dma_cache_alignment(); =20 riscv_user_isa_enable(); + riscv_spinlock_init(); } =20 bool arch_cpu_is_hotpluggable(int cpu) diff --git a/include/asm-generic/qspinlock.h b/include/asm-generic/qspinloc= k.h index 0655aa5b57b2..bf47cca2c375 100644 --- a/include/asm-generic/qspinlock.h +++ b/include/asm-generic/qspinlock.h @@ -136,6 +136,7 @@ static __always_inline bool virt_spin_lock(struct qspin= lock *lock) } #endif =20 +#ifndef __no_arch_spinlock_redefine /* * Remapping spinlock architecture specific functions to the corresponding * queued spinlock functions. @@ -146,5 +147,6 @@ static __always_inline bool virt_spin_lock(struct qspin= lock *lock) #define arch_spin_lock(l) queued_spin_lock(l) #define arch_spin_trylock(l) queued_spin_trylock(l) #define arch_spin_unlock(l) queued_spin_unlock(l) +#endif =20 #endif /* __ASM_GENERIC_QSPINLOCK_H */ diff --git a/include/asm-generic/ticket_spinlock.h b/include/asm-generic/ti= cket_spinlock.h index cfcff22b37b3..325779970d8a 100644 --- a/include/asm-generic/ticket_spinlock.h +++ b/include/asm-generic/ticket_spinlock.h @@ -89,6 +89,7 @@ static __always_inline int ticket_spin_is_contended(arch_= spinlock_t *lock) return (s16)((val >> 16) - (val & 0xffff)) > 1; } =20 +#ifndef __no_arch_spinlock_redefine /* * Remapping spinlock architecture specific functions to the corresponding * ticket spinlock functions. @@ -99,5 +100,6 @@ static __always_inline int ticket_spin_is_contended(arch= _spinlock_t *lock) #define arch_spin_lock(l) ticket_spin_lock(l) #define arch_spin_trylock(l) ticket_spin_trylock(l) #define arch_spin_unlock(l) ticket_spin_unlock(l) +#endif =20 #endif /* __ASM_GENERIC_TICKET_SPINLOCK_H */ --=20 2.39.2