From nobody Fri Dec 26 15:22:09 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6FA2A1C283 for ; Wed, 3 Jan 2024 16:32:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="dJecUMSG" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1704299560; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gfK6+G/fxpQtUcPr0UJaWEhuvxeP6ij0m8scDlFf7oE=; b=dJecUMSGYYED6Ub99gP3RG2KlWmVOZReXJY2YyRZCgPMeDjh7gu4ugkt5cI4ggzZ5wpMN9 KWAIbdLLa6FZ237jQ31mZ5ePOtdDSPnPwM3/cBASG4h8ZcUgCIQ3C/KkU7h4ahmSoolxRF AynmKqHjBB++Q3oyImnVA3+P5LrTkeA= Received: from mail-pj1-f71.google.com (mail-pj1-f71.google.com [209.85.216.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-176-KjXt99EvN_yqnemIdH6GlA-1; Wed, 03 Jan 2024 11:32:38 -0500 X-MC-Unique: KjXt99EvN_yqnemIdH6GlA-1 Received: by mail-pj1-f71.google.com with SMTP id 98e67ed59e1d1-28cb44ab1cbso2281896a91.3 for ; Wed, 03 Jan 2024 08:32:38 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704299558; x=1704904358; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gfK6+G/fxpQtUcPr0UJaWEhuvxeP6ij0m8scDlFf7oE=; b=iUrjNK0ZWdh2aBVdyJlvXRDpJgzO0mbBzly0wWIqFaXi6h5462XN8sxHjPbxa0cmml 7rPq49qTmGwuA3kmWpFv7he5TTPww6vOYKN+IjZfArAt1vIErrM6P0OfCem/cFwem+x2 Rab2777frIdfVHNUxstboXRoRpwJAOb+FMVnC+G8/eLzYrhUCrC1LlB9aqasgW0ganvJ P/cn5bcE3mxep8z9WW2yr4CVdPBIHJo+OtI+xAfLM5xEBH0yOCp/UTPiuNHkacK9VRSg CnIDJmp6JnACUpCDS81Se1FLMDOyQSaIqtwqJx2SWBAF6gfEdecCiyYlcUfTLMmgHr/y 5F9g== X-Gm-Message-State: AOJu0YwDUsrd6lrlL1ve3Biy4zWuNo0Mq3uJ4TjmHNFzKEZbq96wKCTH nWMUiU9dTroK6ugHo3DXtVBjrilk9/v2rlE+z1fyWKbpXlk0QPd6BY2+R1CWE//A0BGry/LX/Bc Lek7qscezH5fp4H7yUyLbYAX1pbfAJyoT X-Received: by 2002:a17:902:f7c9:b0:1d4:79b5:b26 with SMTP id h9-20020a170902f7c900b001d479b50b26mr6275120plw.78.1704299557738; Wed, 03 Jan 2024 08:32:37 -0800 (PST) X-Google-Smtp-Source: AGHT+IFTQgC8TXArNxt0zw+/Ybt1FA1l3Pis1O2gQ6NIBPnJnz+swVe6KE7mMi1vmxykVrwzFBJixQ== X-Received: by 2002:a17:902:f7c9:b0:1d4:79b5:b26 with SMTP id h9-20020a170902f7c900b001d479b50b26mr6275100plw.78.1704299557400; Wed, 03 Jan 2024 08:32:37 -0800 (PST) Received: from localhost.localdomain ([2804:431:c7ec:911:6911:ca60:846:eb46]) by smtp.gmail.com with ESMTPSA id e12-20020a170902b78c00b001cfca7b8ee7sm23930425pls.99.2024.01.03.08.32.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 08:32:36 -0800 (PST) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Guo Ren , Andrea Parri , Geert Uytterhoeven , Ingo Molnar , Andrzej Hajda Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [PATCH v1 1/5] riscv/cmpxchg: Deduplicate xchg() asm functions Date: Wed, 3 Jan 2024 13:31:59 -0300 Message-ID: <20240103163203.72768-3-leobras@redhat.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240103163203.72768-2-leobras@redhat.com> References: <20240103163203.72768-2-leobras@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In this header every xchg define (_relaxed, _acquire, _release, vanilla) contain it's own asm file, both for 4-byte variables an 8-byte variables, on a total of 8 versions of mostly the same asm. This is usually bad, as it means any change may be done in up to 8 different places. Unify those versions by creating a new define with enough parameters to generate any version of the previous 8. Then unify the result under a more general define, and simplify arch_xchg* generation. (This did not cause any change in generated asm) Signed-off-by: Leonardo Bras Reviewed-by: Guo Ren Reviewed-by: Andrea Parri Tested-by: Guo Ren --- arch/riscv/include/asm/cmpxchg.h | 138 ++++++------------------------- 1 file changed, 23 insertions(+), 115 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index 2f4726d3cfcc2..48478a8eecee7 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -11,140 +11,48 @@ #include #include =20 -#define __xchg_relaxed(ptr, new, size) \ +#define __arch_xchg(sfx, prepend, append, r, p, n) \ ({ \ - __typeof__(ptr) __ptr =3D (ptr); \ - __typeof__(new) __new =3D (new); \ - __typeof__(*(ptr)) __ret; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - " amoswap.w %0, %2, %1\n" \ - : "=3Dr" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - " amoswap.d %0, %2, %1\n" \ - : "=3Dr" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) - -#define arch_xchg_relaxed(ptr, x) \ -({ \ - __typeof__(*(ptr)) _x_ =3D (x); \ - (__typeof__(*(ptr))) __xchg_relaxed((ptr), \ - _x_, sizeof(*(ptr))); \ + __asm__ __volatile__ ( \ + prepend \ + " amoswap" sfx " %0, %2, %1\n" \ + append \ + : "=3Dr" (r), "+A" (*(p)) \ + : "r" (n) \ + : "memory"); \ }) =20 -#define __xchg_acquire(ptr, new, size) \ +#define _arch_xchg(ptr, new, sfx, prepend, append) \ ({ \ __typeof__(ptr) __ptr =3D (ptr); \ - __typeof__(new) __new =3D (new); \ - __typeof__(*(ptr)) __ret; \ - switch (size) { \ + __typeof__(*(__ptr)) __new =3D (new); \ + __typeof__(*(__ptr)) __ret; \ + switch (sizeof(*__ptr)) { \ case 4: \ - __asm__ __volatile__ ( \ - " amoswap.w %0, %2, %1\n" \ - RISCV_ACQUIRE_BARRIER \ - : "=3Dr" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ + __arch_xchg(".w" sfx, prepend, append, \ + __ret, __ptr, __new); \ break; \ case 8: \ - __asm__ __volatile__ ( \ - " amoswap.d %0, %2, %1\n" \ - RISCV_ACQUIRE_BARRIER \ - : "=3Dr" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ + __arch_xchg(".d" sfx, prepend, append, \ + __ret, __ptr, __new); \ break; \ default: \ BUILD_BUG(); \ } \ - __ret; \ + (__typeof__(*(__ptr)))__ret; \ }) =20 -#define arch_xchg_acquire(ptr, x) \ -({ \ - __typeof__(*(ptr)) _x_ =3D (x); \ - (__typeof__(*(ptr))) __xchg_acquire((ptr), \ - _x_, sizeof(*(ptr))); \ -}) +#define arch_xchg_relaxed(ptr, x) \ + _arch_xchg(ptr, x, "", "", "") =20 -#define __xchg_release(ptr, new, size) \ -({ \ - __typeof__(ptr) __ptr =3D (ptr); \ - __typeof__(new) __new =3D (new); \ - __typeof__(*(ptr)) __ret; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - " amoswap.w %0, %2, %1\n" \ - : "=3Dr" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - " amoswap.d %0, %2, %1\n" \ - : "=3Dr" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) +#define arch_xchg_acquire(ptr, x) \ + _arch_xchg(ptr, x, "", "", RISCV_ACQUIRE_BARRIER) =20 #define arch_xchg_release(ptr, x) \ -({ \ - __typeof__(*(ptr)) _x_ =3D (x); \ - (__typeof__(*(ptr))) __xchg_release((ptr), \ - _x_, sizeof(*(ptr))); \ -}) - -#define __arch_xchg(ptr, new, size) \ -({ \ - __typeof__(ptr) __ptr =3D (ptr); \ - __typeof__(new) __new =3D (new); \ - __typeof__(*(ptr)) __ret; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - " amoswap.w.aqrl %0, %2, %1\n" \ - : "=3Dr" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - " amoswap.d.aqrl %0, %2, %1\n" \ - : "=3Dr" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) + _arch_xchg(ptr, x, "", RISCV_RELEASE_BARRIER, "") =20 #define arch_xchg(ptr, x) \ -({ \ - __typeof__(*(ptr)) _x_ =3D (x); \ - (__typeof__(*(ptr))) __arch_xchg((ptr), _x_, sizeof(*(ptr))); \ -}) + _arch_xchg(ptr, x, ".aqrl", "", "") =20 #define xchg32(ptr, x) \ ({ \ --=20 2.43.0 From nobody Fri Dec 26 15:22:09 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 89B3F1C693 for ; Wed, 3 Jan 2024 16:32:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="e5wX4zr7" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1704299566; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=n/sNmacMSvCkIR374aY0S9uvqtL96KdIcIOWwUR3yYE=; b=e5wX4zr74cnSpHisEyX80UxPFUYycrUWb3Z7yZulC9Xk/VybiUnJRNfofIh6F8N+sTfb9Q rIvxMkk5TCBoUyiUODy78RZvfIQapN9U8WuiM0NBKtg2zjAoU+yrV6J7DfC7vuOVmHqhSy hX0l7jhElbYw74m0FosisPEr1fl/Sys= Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-594-_r9DBviyMvC-QBE9b1e_Ew-1; Wed, 03 Jan 2024 11:32:45 -0500 X-MC-Unique: _r9DBviyMvC-QBE9b1e_Ew-1 Received: by mail-pg1-f197.google.com with SMTP id 41be03b00d2f7-5ce632b2adfso4302804a12.0 for ; Wed, 03 Jan 2024 08:32:44 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704299563; x=1704904363; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=n/sNmacMSvCkIR374aY0S9uvqtL96KdIcIOWwUR3yYE=; b=L56TUozV45UsNHhQzQSC5AOuWy1Ia4QzlHkGDO09nPRkHJRV2AaYgFtnJgJ+EgUsU2 WUpB/foamYEvd4bJyPdvBmArPBYyEnrlH61iErhpPfwzecb/z+VQj4+03nEc6MZri2VC l1eKgNsTH4Ah9oHL5rCqGcHZ9nUfkGSz25SIZ/+J2qaIh8oBnRfvPcOkhz2t+4ZmTgvf NCcRQoLvRLr6Si4bBdqT/t4usuvAN5abxx9y4eSoB+ufKduHyKb7L6gzveRWTz4iQvAi oWU+JSBGi8vlucCrs/VxJeTbaftTqtlf2onrVOukDOCPHpoUtYJWzoGKUQzDrwydZD/Y 9DbQ== X-Gm-Message-State: AOJu0Yyp5eow9g+SVtgfW4M+RLoTLz524ZpH91OhgZahwf95swa69NUX IJ6cihyCAhAfmOVxoaT2NwgrNplR4ISvHbeH0cibV6SOTpqSQAA5aEw6ejRVeV9OHK5RfCoWOyF HS5WFOVP21Upvgmm5AHtTHNUgG8TZpmRr X-Received: by 2002:a17:902:64c2:b0:1d4:87c8:40f9 with SMTP id y2-20020a17090264c200b001d487c840f9mr10868120pli.84.1704299563376; Wed, 03 Jan 2024 08:32:43 -0800 (PST) X-Google-Smtp-Source: AGHT+IEk9KLzHdKrTHSS/KtZZppx+0qxHeyY2aaqvlMhT293khOuYzfSNdLjxjc3JpSL3I6Ppc3weQ== X-Received: by 2002:a17:902:64c2:b0:1d4:87c8:40f9 with SMTP id y2-20020a17090264c200b001d487c840f9mr10868097pli.84.1704299563019; Wed, 03 Jan 2024 08:32:43 -0800 (PST) Received: from localhost.localdomain ([2804:431:c7ec:911:6911:ca60:846:eb46]) by smtp.gmail.com with ESMTPSA id e12-20020a170902b78c00b001cfca7b8ee7sm23930425pls.99.2024.01.03.08.32.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 08:32:42 -0800 (PST) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Guo Ren , Andrea Parri , Geert Uytterhoeven , Ingo Molnar , Andrzej Hajda Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [PATCH v1 2/5] riscv/cmpxchg: Deduplicate cmpxchg() asm and macros Date: Wed, 3 Jan 2024 13:32:00 -0300 Message-ID: <20240103163203.72768-4-leobras@redhat.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240103163203.72768-2-leobras@redhat.com> References: <20240103163203.72768-2-leobras@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In this header every cmpxchg define (_relaxed, _acquire, _release, vanilla) contain it's own asm file, both for 4-byte variables an 8-byte variables, on a total of 8 versions of mostly the same asm. This is usually bad, as it means any change may be done in up to 8 different places. Unify those versions by creating a new define with enough parameters to generate any version of the previous 8. Then unify the result under a more general define, and simplify arch_cmpxchg* generation (This did not cause any change in generated asm) Signed-off-by: Leonardo Bras Reviewed-by: Guo Ren Reviewed-by: Andrea Parri Tested-by: Guo Ren --- arch/riscv/include/asm/cmpxchg.h | 195 ++++++------------------------- 1 file changed, 33 insertions(+), 162 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index 48478a8eecee7..e3e0ac7ba061b 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -71,190 +71,61 @@ * store NEW in MEM. Return the initial value in MEM. Success is * indicated by comparing RETURN with OLD. */ -#define __cmpxchg_relaxed(ptr, old, new, size) \ -({ \ - __typeof__(ptr) __ptr =3D (ptr); \ - __typeof__(*(ptr)) __old =3D (old); \ - __typeof__(*(ptr)) __new =3D (new); \ - __typeof__(*(ptr)) __ret; \ - register unsigned int __rc; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - "0: lr.w %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.w %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - "1:\n" \ - : "=3D&r" (__ret), "=3D&r" (__rc), "+A" (*__ptr) \ - : "rJ" ((long)__old), "rJ" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - "0: lr.d %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.d %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - "1:\n" \ - : "=3D&r" (__ret), "=3D&r" (__rc), "+A" (*__ptr) \ - : "rJ" (__old), "rJ" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) =20 -#define arch_cmpxchg_relaxed(ptr, o, n) \ -({ \ - __typeof__(*(ptr)) _o_ =3D (o); \ - __typeof__(*(ptr)) _n_ =3D (n); \ - (__typeof__(*(ptr))) __cmpxchg_relaxed((ptr), \ - _o_, _n_, sizeof(*(ptr))); \ -}) =20 -#define __cmpxchg_acquire(ptr, old, new, size) \ +#define __arch_cmpxchg(lr_sfx, sc_sfx, prepend, append, r, p, co, o, n) \ ({ \ - __typeof__(ptr) __ptr =3D (ptr); \ - __typeof__(*(ptr)) __old =3D (old); \ - __typeof__(*(ptr)) __new =3D (new); \ - __typeof__(*(ptr)) __ret; \ register unsigned int __rc; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - "0: lr.w %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.w %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - RISCV_ACQUIRE_BARRIER \ - "1:\n" \ - : "=3D&r" (__ret), "=3D&r" (__rc), "+A" (*__ptr) \ - : "rJ" ((long)__old), "rJ" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - "0: lr.d %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.d %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - RISCV_ACQUIRE_BARRIER \ - "1:\n" \ - : "=3D&r" (__ret), "=3D&r" (__rc), "+A" (*__ptr) \ - : "rJ" (__old), "rJ" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) - -#define arch_cmpxchg_acquire(ptr, o, n) \ -({ \ - __typeof__(*(ptr)) _o_ =3D (o); \ - __typeof__(*(ptr)) _n_ =3D (n); \ - (__typeof__(*(ptr))) __cmpxchg_acquire((ptr), \ - _o_, _n_, sizeof(*(ptr))); \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr" lr_sfx " %0, %2\n" \ + " bne %0, %z3, 1f\n" \ + " sc" sc_sfx " %1, %z4, %2\n" \ + " bnez %1, 0b\n" \ + append \ + "1:\n" \ + : "=3D&r" (r), "=3D&r" (__rc), "+A" (*(p)) \ + : "rJ" (co o), "rJ" (n) \ + : "memory"); \ }) =20 -#define __cmpxchg_release(ptr, old, new, size) \ +#define _arch_cmpxchg(ptr, old, new, sc_sfx, prepend, append) \ ({ \ __typeof__(ptr) __ptr =3D (ptr); \ - __typeof__(*(ptr)) __old =3D (old); \ - __typeof__(*(ptr)) __new =3D (new); \ - __typeof__(*(ptr)) __ret; \ - register unsigned int __rc; \ - switch (size) { \ + __typeof__(*(__ptr)) __old =3D (old); \ + __typeof__(*(__ptr)) __new =3D (new); \ + __typeof__(*(__ptr)) __ret; \ + \ + switch (sizeof(*__ptr)) { \ case 4: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - "0: lr.w %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.w %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - "1:\n" \ - : "=3D&r" (__ret), "=3D&r" (__rc), "+A" (*__ptr) \ - : "rJ" ((long)__old), "rJ" (__new) \ - : "memory"); \ + __arch_cmpxchg(".w", ".w" sc_sfx, prepend, append, \ + __ret, __ptr, (long), __old, __new); \ break; \ case 8: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - "0: lr.d %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.d %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - "1:\n" \ - : "=3D&r" (__ret), "=3D&r" (__rc), "+A" (*__ptr) \ - : "rJ" (__old), "rJ" (__new) \ - : "memory"); \ + __arch_cmpxchg(".d", ".d" sc_sfx, prepend, append, \ + __ret, __ptr, /**/, __old, __new); \ break; \ default: \ BUILD_BUG(); \ } \ - __ret; \ + (__typeof__(*(__ptr)))__ret; \ }) =20 -#define arch_cmpxchg_release(ptr, o, n) \ -({ \ - __typeof__(*(ptr)) _o_ =3D (o); \ - __typeof__(*(ptr)) _n_ =3D (n); \ - (__typeof__(*(ptr))) __cmpxchg_release((ptr), \ - _o_, _n_, sizeof(*(ptr))); \ -}) +#define arch_cmpxchg_relaxed(ptr, o, n) \ + _arch_cmpxchg((ptr), (o), (n), "", "", "") =20 -#define __cmpxchg(ptr, old, new, size) \ -({ \ - __typeof__(ptr) __ptr =3D (ptr); \ - __typeof__(*(ptr)) __old =3D (old); \ - __typeof__(*(ptr)) __new =3D (new); \ - __typeof__(*(ptr)) __ret; \ - register unsigned int __rc; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - "0: lr.w %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.w.rl %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - " fence rw, rw\n" \ - "1:\n" \ - : "=3D&r" (__ret), "=3D&r" (__rc), "+A" (*__ptr) \ - : "rJ" ((long)__old), "rJ" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - "0: lr.d %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.d.rl %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - " fence rw, rw\n" \ - "1:\n" \ - : "=3D&r" (__ret), "=3D&r" (__rc), "+A" (*__ptr) \ - : "rJ" (__old), "rJ" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) +#define arch_cmpxchg_acquire(ptr, o, n) \ + _arch_cmpxchg((ptr), (o), (n), "", "", RISCV_ACQUIRE_BARRIER) + +#define arch_cmpxchg_release(ptr, o, n) \ + _arch_cmpxchg((ptr), (o), (n), "", RISCV_RELEASE_BARRIER, "") =20 #define arch_cmpxchg(ptr, o, n) \ -({ \ - __typeof__(*(ptr)) _o_ =3D (o); \ - __typeof__(*(ptr)) _n_ =3D (n); \ - (__typeof__(*(ptr))) __cmpxchg((ptr), \ - _o_, _n_, sizeof(*(ptr))); \ -}) + _arch_cmpxchg((ptr), (o), (n), ".rl", "", " fence rw, rw\n") =20 #define arch_cmpxchg_local(ptr, o, n) \ - (__cmpxchg_relaxed((ptr), (o), (n), sizeof(*(ptr)))) + arch_cmpxchg_relaxed((ptr), (o), (n)) =20 #define arch_cmpxchg64(ptr, o, n) \ ({ \ --=20 2.43.0 From nobody Fri Dec 26 15:22:09 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5A0191CA85 for ; Wed, 3 Jan 2024 16:32:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="PU+IGV1A" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1704299578; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DNxCEyFjygCY2k/IIyJiEePHcERbzG2CmUQITkDIUjg=; b=PU+IGV1A75Te7BOue3fGa2OYJzVfSwDipPgbMxLbG15Q00ipprHVKpNk/n8vKNzhyejgkl Pf28c2zPk4toDYfh9E7AgPRsVTLJOkDWYDf020SUhd2MgujoGoeb4lCgavpJjlr4Gwtaq/ h1QJWrxkWuETPbIoJFywvqYVNKL81bo= Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-296-kibDYC4qOfKn_voFnZA2QA-1; Wed, 03 Jan 2024 11:32:49 -0500 X-MC-Unique: kibDYC4qOfKn_voFnZA2QA-1 Received: by mail-pl1-f197.google.com with SMTP id d9443c01a7336-1d3eaaaa5a1so4722205ad.1 for ; Wed, 03 Jan 2024 08:32:49 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704299568; x=1704904368; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DNxCEyFjygCY2k/IIyJiEePHcERbzG2CmUQITkDIUjg=; b=T8kHsp6kg+9/vutkyFVIxuM6L4YmsmNIEOBvYwbkMKqu78h7bI3wDSrPj2Fjh+eZpR ulw6I7xtJnCSjXExCDzSZ4Gyji21d+9N9TN7crMhvogWY5cqVMLRVpSR4k/iAp/c8IMG jd6dp3fPR4kEVSneyAuZn0ABCXMJREO7QIMrQloz1hGt+zYwle8nWTS8TNNJJ2zjbgUP dUVBOBVRxRCzHlLB6MJqWSRJQLMcG9utKKPoYNRMV+QCV57h/QSiBaMvhNG1rFZDevwF 5C9YVapMMb87QqS0G3P3RCsSSrBxi526pwK7jJ6bMVHObmbv25mVWY6BCRUV/3atVknh IY0g== X-Gm-Message-State: AOJu0YyC7GdUhSPUJg4Mq9lafk/b4uhyHu780RQHRl5BPZk5jSLFDi1u ry6nPfjMojzb/05xJTGxZECqfSlIUhZX9X7EhRpiZ8c60XQrThpXW47QwK2IujDEfm/gqfvr8LV LY2b3YsWSap4khfDdH74rFIjiJWozl2pu X-Received: by 2002:a17:903:cb:b0:1d4:94f7:b349 with SMTP id x11-20020a17090300cb00b001d494f7b349mr1393333plc.54.1704299568593; Wed, 03 Jan 2024 08:32:48 -0800 (PST) X-Google-Smtp-Source: AGHT+IGRh2QWtQD29WPJYeQ0K2F3ztPLRvH+XRo73kfZaTuSm2TpoOzI5zg6Q9eJt87gGsc7c2/5DA== X-Received: by 2002:a17:903:cb:b0:1d4:94f7:b349 with SMTP id x11-20020a17090300cb00b001d494f7b349mr1393323plc.54.1704299568269; Wed, 03 Jan 2024 08:32:48 -0800 (PST) Received: from localhost.localdomain ([2804:431:c7ec:911:6911:ca60:846:eb46]) by smtp.gmail.com with ESMTPSA id e12-20020a170902b78c00b001cfca7b8ee7sm23930425pls.99.2024.01.03.08.32.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 08:32:47 -0800 (PST) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Guo Ren , Andrea Parri , Geert Uytterhoeven , Ingo Molnar , Andrzej Hajda Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [PATCH v1 3/5] riscv/atomic.h : Deduplicate arch_atomic.* Date: Wed, 3 Jan 2024 13:32:01 -0300 Message-ID: <20240103163203.72768-5-leobras@redhat.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240103163203.72768-2-leobras@redhat.com> References: <20240103163203.72768-2-leobras@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Some functions use mostly the same asm for 32-bit and 64-bit versions. Make a macro that is generic enough and avoid code duplication. (This did not cause any change in generated asm) Signed-off-by: Leonardo Bras Reviewed-by: Guo Ren Reviewed-by: Andrea Parri Tested-by: Guo Ren --- arch/riscv/include/asm/atomic.h | 164 +++++++++++++++----------------- 1 file changed, 76 insertions(+), 88 deletions(-) diff --git a/arch/riscv/include/asm/atomic.h b/arch/riscv/include/asm/atomi= c.h index f5dfef6c2153f..80cca7ac16fd3 100644 --- a/arch/riscv/include/asm/atomic.h +++ b/arch/riscv/include/asm/atomic.h @@ -196,22 +196,28 @@ ATOMIC_OPS(xor, xor, i) #undef ATOMIC_FETCH_OP #undef ATOMIC_OP_RETURN =20 +#define _arch_atomic_fetch_add_unless(_prev, _rc, counter, _a, _u, sfx) \ +({ \ + __asm__ __volatile__ ( \ + "0: lr." sfx " %[p], %[c]\n" \ + " beq %[p], %[u], 1f\n" \ + " add %[rc], %[p], %[a]\n" \ + " sc." sfx ".rl %[rc], %[rc], %[c]\n" \ + " bnez %[rc], 0b\n" \ + " fence rw, rw\n" \ + "1:\n" \ + : [p]"=3D&r" (_prev), [rc]"=3D&r" (_rc), [c]"+A" (counter) \ + : [a]"r" (_a), [u]"r" (_u) \ + : "memory"); \ +}) + /* This is required to provide a full barrier on success. */ static __always_inline int arch_atomic_fetch_add_unless(atomic_t *v, int a= , int u) { int prev, rc; =20 - __asm__ __volatile__ ( - "0: lr.w %[p], %[c]\n" - " beq %[p], %[u], 1f\n" - " add %[rc], %[p], %[a]\n" - " sc.w.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=3D&r" (prev), [rc]"=3D&r" (rc), [c]"+A" (v->counter) - : [a]"r" (a), [u]"r" (u) - : "memory"); + _arch_atomic_fetch_add_unless(prev, rc, v->counter, a, u, "w"); + return prev; } #define arch_atomic_fetch_add_unless arch_atomic_fetch_add_unless @@ -222,77 +228,86 @@ static __always_inline s64 arch_atomic64_fetch_add_un= less(atomic64_t *v, s64 a, s64 prev; long rc; =20 - __asm__ __volatile__ ( - "0: lr.d %[p], %[c]\n" - " beq %[p], %[u], 1f\n" - " add %[rc], %[p], %[a]\n" - " sc.d.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=3D&r" (prev), [rc]"=3D&r" (rc), [c]"+A" (v->counter) - : [a]"r" (a), [u]"r" (u) - : "memory"); + _arch_atomic_fetch_add_unless(prev, rc, v->counter, a, u, "d"); + return prev; } #define arch_atomic64_fetch_add_unless arch_atomic64_fetch_add_unless #endif =20 +#define _arch_atomic_inc_unless_negative(_prev, _rc, counter, sfx) \ +({ \ + __asm__ __volatile__ ( \ + "0: lr." sfx " %[p], %[c]\n" \ + " bltz %[p], 1f\n" \ + " addi %[rc], %[p], 1\n" \ + " sc." sfx ".rl %[rc], %[rc], %[c]\n" \ + " bnez %[rc], 0b\n" \ + " fence rw, rw\n" \ + "1:\n" \ + : [p]"=3D&r" (_prev), [rc]"=3D&r" (_rc), [c]"+A" (counter) \ + : \ + : "memory"); \ +}) + static __always_inline bool arch_atomic_inc_unless_negative(atomic_t *v) { int prev, rc; =20 - __asm__ __volatile__ ( - "0: lr.w %[p], %[c]\n" - " bltz %[p], 1f\n" - " addi %[rc], %[p], 1\n" - " sc.w.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=3D&r" (prev), [rc]"=3D&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_inc_unless_negative(prev, rc, v->counter, "w"); + return !(prev < 0); } =20 #define arch_atomic_inc_unless_negative arch_atomic_inc_unless_negative =20 +#define _arch_atomic_dec_unless_positive(_prev, _rc, counter, sfx) \ +({ \ + __asm__ __volatile__ ( \ + "0: lr." sfx " %[p], %[c]\n" \ + " bgtz %[p], 1f\n" \ + " addi %[rc], %[p], -1\n" \ + " sc." sfx ".rl %[rc], %[rc], %[c]\n" \ + " bnez %[rc], 0b\n" \ + " fence rw, rw\n" \ + "1:\n" \ + : [p]"=3D&r" (_prev), [rc]"=3D&r" (_rc), [c]"+A" (counter) \ + : \ + : "memory"); \ +}) + static __always_inline bool arch_atomic_dec_unless_positive(atomic_t *v) { int prev, rc; =20 - __asm__ __volatile__ ( - "0: lr.w %[p], %[c]\n" - " bgtz %[p], 1f\n" - " addi %[rc], %[p], -1\n" - " sc.w.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=3D&r" (prev), [rc]"=3D&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_dec_unless_positive(prev, rc, v->counter, "w"); + return !(prev > 0); } =20 #define arch_atomic_dec_unless_positive arch_atomic_dec_unless_positive =20 +#define _arch_atomic_dec_if_positive(_prev, _rc, counter, sfx) \ +({ \ + __asm__ __volatile__ ( \ + "0: lr." sfx " %[p], %[c]\n" \ + " addi %[rc], %[p], -1\n" \ + " bltz %[rc], 1f\n" \ + " sc." sfx ".rl %[rc], %[rc], %[c]\n" \ + " bnez %[rc], 0b\n" \ + " fence rw, rw\n" \ + "1:\n" \ + : [p]"=3D&r" (_prev), [rc]"=3D&r" (_rc), [c]"+A" (counter) \ + : \ + : "memory"); \ +}) + static __always_inline int arch_atomic_dec_if_positive(atomic_t *v) { int prev, rc; =20 - __asm__ __volatile__ ( - "0: lr.w %[p], %[c]\n" - " addi %[rc], %[p], -1\n" - " bltz %[rc], 1f\n" - " sc.w.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=3D&r" (prev), [rc]"=3D&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_dec_if_positive(prev, rc, v->counter, "w"); + return prev - 1; } =20 @@ -304,17 +319,8 @@ static __always_inline bool arch_atomic64_inc_unless_n= egative(atomic64_t *v) s64 prev; long rc; =20 - __asm__ __volatile__ ( - "0: lr.d %[p], %[c]\n" - " bltz %[p], 1f\n" - " addi %[rc], %[p], 1\n" - " sc.d.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=3D&r" (prev), [rc]"=3D&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_inc_unless_negative(prev, rc, v->counter, "d"); + return !(prev < 0); } =20 @@ -325,17 +331,8 @@ static __always_inline bool arch_atomic64_dec_unless_p= ositive(atomic64_t *v) s64 prev; long rc; =20 - __asm__ __volatile__ ( - "0: lr.d %[p], %[c]\n" - " bgtz %[p], 1f\n" - " addi %[rc], %[p], -1\n" - " sc.d.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=3D&r" (prev), [rc]"=3D&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_dec_unless_positive(prev, rc, v->counter, "d"); + return !(prev > 0); } =20 @@ -346,17 +343,8 @@ static __always_inline s64 arch_atomic64_dec_if_positi= ve(atomic64_t *v) s64 prev; long rc; =20 - __asm__ __volatile__ ( - "0: lr.d %[p], %[c]\n" - " addi %[rc], %[p], -1\n" - " bltz %[rc], 1f\n" - " sc.d.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=3D&r" (prev), [rc]"=3D&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_dec_if_positive(prev, rc, v->counter, "d"); + return prev - 1; } =20 --=20 2.43.0 From nobody Fri Dec 26 15:22:09 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DD8F61CA81 for ; Wed, 3 Jan 2024 16:32:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="hBVV5zNH" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1704299576; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Zrl9wQMm72/9gkP6lQrsUXatEy2G8nXg2opkxwrV5kI=; b=hBVV5zNH5bcUvVUr78BN0rJ6sSpsIXkfNcbBRCLsNKhYL50kx6xLl77Z2FS17lPx80BLSp k4hh6FrYR9NFcRXDNSpbL9v3IfrhE0HdOA5CKpnqMOmuj1SliVtddEhzipx0/A2fybV3UM JRawnwTuE7a4u0RBuet29qmiYTYNkgo= Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-192-tx-pRf50NYy0reYBYCI21g-1; Wed, 03 Jan 2024 11:32:55 -0500 X-MC-Unique: tx-pRf50NYy0reYBYCI21g-1 Received: by mail-pl1-f197.google.com with SMTP id d9443c01a7336-1d4bc42624eso23481355ad.1 for ; Wed, 03 Jan 2024 08:32:55 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704299574; x=1704904374; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Zrl9wQMm72/9gkP6lQrsUXatEy2G8nXg2opkxwrV5kI=; b=Uf800widcdWGM/e8KiMavEDaacqWQ3XPu971UpDSmGrx6Wf4sSellj+mVv9+iWZSYF VeIFiBxQkydsobCv4OIqYxCt8HTmBGjFMpTqKwysmlZMlhTsU1tXACfCTeYyF4CoMQ0t VXeLd5ixzU6VA6F02+dJpz7hVLI5sYjiWlENo0msmtciwuPQbqeDeCnBPh7WmotGYtZk G29BjQM6jQyZMkDX4niN6LdrUZJrRhfDqjL7VyY7XHFTMv7t/sxcFN5GVLu5hCJMMbAa EjvYV5d09p4T/wp+vE/Vo7V8VXLF3v1UoY7K4OnlhWrtzFyQIZg/gsy0MMrRrX4kvN/k PrdA== X-Gm-Message-State: AOJu0Yw65FAXAxFTS0xqc/sF3MqHdfrakF2ZU5iF7ylzDHEdbViPkJvZ xLCZsrddxsEKtxhlXc9w81218ZHKIzM6+qAq29ZlyGH10serR82uU+qp66q4/mtuAlE7cgrAK81 Nnz3MkoJ2ZBikk6E3UZA8MFeapVey/EPJPZohmiKg X-Received: by 2002:a17:902:b213:b0:1d4:3dfd:7e31 with SMTP id t19-20020a170902b21300b001d43dfd7e31mr20225789plr.139.1704299573989; Wed, 03 Jan 2024 08:32:53 -0800 (PST) X-Google-Smtp-Source: AGHT+IFuxxr3jzBnWJLb5k5Ijiau2Uo52Uln5XHUYl8Xl2mWgyNYjwzpHX8ACzhU950ngT5irx/v3w== X-Received: by 2002:a17:902:b213:b0:1d4:3dfd:7e31 with SMTP id t19-20020a170902b21300b001d43dfd7e31mr20225778plr.139.1704299573733; Wed, 03 Jan 2024 08:32:53 -0800 (PST) Received: from localhost.localdomain ([2804:431:c7ec:911:6911:ca60:846:eb46]) by smtp.gmail.com with ESMTPSA id e12-20020a170902b78c00b001cfca7b8ee7sm23930425pls.99.2024.01.03.08.32.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 08:32:53 -0800 (PST) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Guo Ren , Andrea Parri , Geert Uytterhoeven , Ingo Molnar , Andrzej Hajda Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [PATCH v1 4/5] riscv/cmpxchg: Implement cmpxchg for variables of size 1 and 2 Date: Wed, 3 Jan 2024 13:32:02 -0300 Message-ID: <20240103163203.72768-6-leobras@redhat.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240103163203.72768-2-leobras@redhat.com> References: <20240103163203.72768-2-leobras@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" cmpxchg for variables of size 1-byte and 2-bytes is not yet available for riscv, even though its present in other architectures such as arm64 and x86. This could lead to not being able to implement some locking mechanisms or requiring some rework to make it work properly. Implement 1-byte and 2-bytes cmpxchg in order to achieve parity with other architectures. Signed-off-by: Leonardo Bras Tested-by: Guo Ren --- arch/riscv/include/asm/cmpxchg.h | 34 ++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index e3e0ac7ba061b..ac9d0eeb74e67 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -72,6 +72,35 @@ * indicated by comparing RETURN with OLD. */ =20 +#define __arch_cmpxchg_masked(sc_sfx, prepend, append, r, p, o, n) \ +({ \ + u32 *__ptr32b =3D (u32 *)((ulong)(p) & ~0x3); \ + ulong __s =3D ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ + ulong __mask =3D GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ + << __s; \ + ulong __newx =3D (ulong)(n) << __s; \ + ulong __oldx =3D (ulong)(o) << __s; \ + ulong __retx; \ + ulong __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr.w %0, %2\n" \ + " and %1, %0, %z5\n" \ + " bne %1, %z3, 1f\n" \ + " and %1, %0, %z6\n" \ + " or %1, %1, %z4\n" \ + " sc.w" sc_sfx " %1, %1, %2\n" \ + " bnez %1, 0b\n" \ + append \ + "1:\n" \ + : "=3D&r" (__retx), "=3D&r" (__rc), "+A" (*(__ptr32b)) \ + : "rJ" ((long)__oldx), "rJ" (__newx), \ + "rJ" (__mask), "rJ" (~__mask) \ + : "memory"); \ + \ + r =3D (__typeof__(*(p)))((__retx & __mask) >> __s); \ +}) =20 #define __arch_cmpxchg(lr_sfx, sc_sfx, prepend, append, r, p, co, o, n) \ ({ \ @@ -98,6 +127,11 @@ __typeof__(*(__ptr)) __ret; \ \ switch (sizeof(*__ptr)) { \ + case 1: \ + case 2: \ + __arch_cmpxchg_masked(sc_sfx, prepend, append, \ + __ret, __ptr, __old, __new); \ + break; \ case 4: \ __arch_cmpxchg(".w", ".w" sc_sfx, prepend, append, \ __ret, __ptr, (long), __old, __new); \ --=20 2.43.0 From nobody Fri Dec 26 15:22:09 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5EE871CF8B for ; Wed, 3 Jan 2024 16:33:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ErWqSiOX" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1704299584; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Re2lmE+u9xJehZpa61KU1PY4uKcHNA9CavGcQorFlps=; b=ErWqSiOXjrIiPgx5l4OBqncVH3pWrzFEwMiwLuUjQva/YO67SgblparFCnw132STPqveXL 6ib+9fvC0WLY4TsmRV1r2zQNylXnro5VvK+At/zkIBUOFgDpXheJYGAi4JvbLHYIH3Gopg IiUvhR8c//IW0oHCNySLDfE2rRFp7L4= Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-612-iLeJwFxfNiuF6opvArAgLw-1; Wed, 03 Jan 2024 11:33:00 -0500 X-MC-Unique: iLeJwFxfNiuF6opvArAgLw-1 Received: by mail-pl1-f200.google.com with SMTP id d9443c01a7336-1d3e5d18308so110242555ad.3 for ; Wed, 03 Jan 2024 08:33:00 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704299579; x=1704904379; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Re2lmE+u9xJehZpa61KU1PY4uKcHNA9CavGcQorFlps=; b=Zt5ZJVcLCNOHiQJP4xvpd56iFqzkmZt1TWRhfRRUyOise1IRllWM3FcXxF0dC3cKTM ZZuv1RXTb9wHPcrJ0IXlmPEZr2pfAOqGyP2rjwTZgugowfKKN5hJ6S3ttYsaSPH1POYC A7KgdH4m6jLztigLjuJLFZbE4O8X1Re4H8vPXZl4/Xx4CxngXwQhg8uFRNGdQ/hivJjK rqAildZSuR9T5J/Yz07/g5Vflli0pmGTA6OQjDyfACXWy14hmpJdrD+v8GlF4nXz+a8w wGMEfhb+AYzE0LpDHvnGTfBmbP6a5RotQ3a+8IPjOghfpKHQbmzLRtiy319PX5wMHK0q zOTQ== X-Gm-Message-State: AOJu0YzNE3MW5o1xlzlgwHzytwGXHD61IJRKypT7xliduS7cK1gsZ64b a6sFZX6jT44R1WQi8P8MGI978sd0OmaTseTIOSNJ959QVjCrWb3wAwmNQnbXpaPVQVNq6KOVvuO OaX8B0b7ElkZ/myDAVvRxH0ghsy0erhED X-Received: by 2002:a17:902:b94c:b0:1d0:6ffe:a11 with SMTP id h12-20020a170902b94c00b001d06ffe0a11mr22000206pls.111.1704299579620; Wed, 03 Jan 2024 08:32:59 -0800 (PST) X-Google-Smtp-Source: AGHT+IFssUiBpWP8x7axGoljRgCI2CzrmRE+huZgi6wtwmSBGSU76WiGWvRbbX36kTz3/VUagQIV+g== X-Received: by 2002:a17:902:b94c:b0:1d0:6ffe:a11 with SMTP id h12-20020a170902b94c00b001d06ffe0a11mr22000194pls.111.1704299579375; Wed, 03 Jan 2024 08:32:59 -0800 (PST) Received: from localhost.localdomain ([2804:431:c7ec:911:6911:ca60:846:eb46]) by smtp.gmail.com with ESMTPSA id e12-20020a170902b78c00b001cfca7b8ee7sm23930425pls.99.2024.01.03.08.32.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 08:32:58 -0800 (PST) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Guo Ren , Andrea Parri , Geert Uytterhoeven , Ingo Molnar , Andrzej Hajda Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [PATCH v1 5/5] riscv/cmpxchg: Implement xchg for variables of size 1 and 2 Date: Wed, 3 Jan 2024 13:32:03 -0300 Message-ID: <20240103163203.72768-7-leobras@redhat.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240103163203.72768-2-leobras@redhat.com> References: <20240103163203.72768-2-leobras@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" xchg for variables of size 1-byte and 2-bytes is not yet available for riscv, even though its present in other architectures such as arm64 and x86. This could lead to not being able to implement some locking mechanisms or requiring some rework to make it work properly. Implement 1-byte and 2-bytes xchg in order to achieve parity with other architectures. Signed-off-by: Leonardo Bras Tested-by: Guo Ren --- arch/riscv/include/asm/cmpxchg.h | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index ac9d0eeb74e67..26cea2395aae8 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -11,6 +11,31 @@ #include #include =20 +#define __arch_xchg_masked(prepend, append, r, p, n) \ +({ \ + u32 *__ptr32b =3D (u32 *)((ulong)(p) & ~0x3); \ + ulong __s =3D ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ + ulong __mask =3D GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ + << __s; \ + ulong __newx =3D (ulong)(n) << __s; \ + ulong __retx; \ + ulong __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr.w %0, %2\n" \ + " and %1, %0, %z4\n" \ + " or %1, %1, %z3\n" \ + " sc.w %1, %1, %2\n" \ + " bnez %1, 0b\n" \ + append \ + : "=3D&r" (__retx), "=3D&r" (__rc), "+A" (*(__ptr32b)) \ + : "rJ" (__newx), "rJ" (~__mask) \ + : "memory"); \ + \ + r =3D (__typeof__(*(p)))((__retx & __mask) >> __s); \ +}) + #define __arch_xchg(sfx, prepend, append, r, p, n) \ ({ \ __asm__ __volatile__ ( \ @@ -27,7 +52,13 @@ __typeof__(ptr) __ptr =3D (ptr); \ __typeof__(*(__ptr)) __new =3D (new); \ __typeof__(*(__ptr)) __ret; \ + \ switch (sizeof(*__ptr)) { \ + case 1: \ + case 2: \ + __arch_xchg_masked(prepend, append, \ + __ret, __ptr, __new); \ + break; \ case 4: \ __arch_xchg(".w" sfx, prepend, append, \ __ret, __ptr, __new); \ --=20 2.43.0