From nobody Tue Apr 14 22:46:25 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5FC6C19F2D for ; Fri, 29 Jul 2022 19:02:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238498AbiG2TCq (ORCPT ); Fri, 29 Jul 2022 15:02:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50868 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238394AbiG2TCo (ORCPT ); Fri, 29 Jul 2022 15:02:44 -0400 Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E3EFB86C29; Fri, 29 Jul 2022 12:02:42 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 85E853187FD; Fri, 29 Jul 2022 15:02:41 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id lZRNnGx2Vadg; Fri, 29 Jul 2022 15:02:41 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 42463318C62; Fri, 29 Jul 2022 15:02:41 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 42463318C62 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1659121361; bh=pVzuqM3AzOqHAueSPhwC/tpBjnInPIjAGgyb3P8mKbE=; h=From:To:Date:Message-Id; b=IWLUHNDa0ARHMb0upK8yadGFtdj49z9UG0xmbIbC3OZctpFsK9jQeUQJLi25zfV5n YrqP5FavcxqA+m5qRbSPtxH7dOZQz4x36xaw/KKR0+BVfSlxN3Cbg67R+PiKU56+PR aJZFePL0f8L7eXD4Y3AWZ1wytZWUnJjoMTYEM8yqDeYL2HPqzUeNOrvZihE73bwUfb djZS3FEOtEde+5jgVvylgieA14T3LMEuqNv2zadE3+4b3V8mbjM9vgipKATLD9K01i H/wQq9sUX/JXg6o0V7VrMtPvW98KyTLndWizxf816+nh9M900CIMdW/GWVIKQgeErB UpqWb2RUT6fSg== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id jmNhjaRfoRE9; Fri, 29 Jul 2022 15:02:41 -0400 (EDT) Received: from localhost.localdomain (192-222-180-24.qc.cable.ebox.net [192.222.180.24]) by mail.efficios.com (Postfix) with ESMTPSA id E6044318F87; Fri, 29 Jul 2022 15:02:40 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Thomas Gleixner , "Paul E . McKenney" , Boqun Feng , "H . Peter Anvin" , Paul Turner , linux-api@vger.kernel.org, Christian Brauner , Florian Weimer , David.Laight@ACULAB.COM, carlos@redhat.com, Peter Oskolkov , Mathieu Desnoyers Subject: [PATCH v3 01/23] rseq: Introduce feature size and alignment ELF auxiliary vector entries Date: Fri, 29 Jul 2022 15:02:03 -0400 Message-Id: <20220729190225.12726-2-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> References: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Export the rseq feature size supported by the kernel as well as the required allocation alignment for the rseq per-thread area to user-space through ELF auxiliary vector entries. This is part of the extensible rseq ABI. Signed-off-by: Mathieu Desnoyers --- fs/binfmt_elf.c | 5 +++++ include/uapi/linux/auxvec.h | 2 ++ include/uapi/linux/rseq.h | 5 +++++ 3 files changed, 12 insertions(+) diff --git a/fs/binfmt_elf.c b/fs/binfmt_elf.c index 63c7ebb0da89..04fca1e4cbd2 100644 --- a/fs/binfmt_elf.c +++ b/fs/binfmt_elf.c @@ -46,6 +46,7 @@ #include #include #include +#include #include #include =20 @@ -288,6 +289,10 @@ create_elf_tables(struct linux_binprm *bprm, const str= uct elfhdr *exec, if (bprm->have_execfd) { NEW_AUX_ENT(AT_EXECFD, bprm->execfd); } +#ifdef CONFIG_RSEQ + NEW_AUX_ENT(AT_RSEQ_FEATURE_SIZE, offsetof(struct rseq, end)); + NEW_AUX_ENT(AT_RSEQ_ALIGN, __alignof__(struct rseq)); +#endif #undef NEW_AUX_ENT /* AT_NULL is zero; clear the rest too */ memset(elf_info, 0, (char *)mm->saved_auxv + diff --git a/include/uapi/linux/auxvec.h b/include/uapi/linux/auxvec.h index c7e502bf5a6f..6991c4b8ab18 100644 --- a/include/uapi/linux/auxvec.h +++ b/include/uapi/linux/auxvec.h @@ -30,6 +30,8 @@ * differ from AT_PLATFORM. */ #define AT_RANDOM 25 /* address of 16 random bytes */ #define AT_HWCAP2 26 /* extension of AT_HWCAP */ +#define AT_RSEQ_FEATURE_SIZE 27 /* rseq supported feature size */ +#define AT_RSEQ_ALIGN 28 /* rseq allocation alignment */ =20 #define AT_EXECFN 31 /* filename of program */ =20 diff --git a/include/uapi/linux/rseq.h b/include/uapi/linux/rseq.h index 77ee207623a9..05d3c4cdeb40 100644 --- a/include/uapi/linux/rseq.h +++ b/include/uapi/linux/rseq.h @@ -130,6 +130,11 @@ struct rseq { * this thread. */ __u32 flags; + + /* + * Flexible array member at end of structure, after last feature field. + */ + char end[]; } __attribute__((aligned(4 * sizeof(__u64)))); =20 #endif /* _UAPI_LINUX_RSEQ_H */ --=20 2.17.1 From nobody Tue Apr 14 22:46:25 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22ECBC00144 for ; Fri, 29 Jul 2022 19:03:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238523AbiG2TC5 (ORCPT ); Fri, 29 Jul 2022 15:02:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50884 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238005AbiG2TCo (ORCPT ); Fri, 29 Jul 2022 15:02:44 -0400 Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E406287373; Fri, 29 Jul 2022 12:02:42 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 154BA318F92; Fri, 29 Jul 2022 15:02:42 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id MnebPTBsrdV9; Fri, 29 Jul 2022 15:02:41 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 8EDF6318B4C; Fri, 29 Jul 2022 15:02:41 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 8EDF6318B4C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1659121361; bh=Cc9JbG7gjTzAaLAreBIoiB9GFv5TVR736Y/5PPIgAkY=; h=From:To:Date:Message-Id; b=o9YC6ZsmQlWvhOyMmpZJG5bAan8PciAivRvOGciiejnE/gICuLfNexskPcCbDhito IqyLHDcL8EdrpVZRlEJzoEkmGkoZPs06dyNSxJ4+YVsU5BrXyCbJtL4dMDiUm95WTa Q0Qmkd/oqT9tnqAOzKczJXyPIu3MDC5ae2AGQLJMEdBfZV+mQ6c3meMFNUKC3iPRn3 Rd+4VP6nxTHApgjXbhZrcKb+gyavsPLJ20Cvgjrsh7vfn7ayc6z+MIMOhcyBx5qrvC C/4qniHbao6oGA5Qn+5xZ0wMhBY4F72Aq7nVe1qFTKiC4N2xuJsBE//7IIcKTfe+cl 2EIk6hoAX49Sg== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 1fYbi2g1nRpQ; Fri, 29 Jul 2022 15:02:41 -0400 (EDT) Received: from localhost.localdomain (192-222-180-24.qc.cable.ebox.net [192.222.180.24]) by mail.efficios.com (Postfix) with ESMTPSA id 450A3318E13; Fri, 29 Jul 2022 15:02:41 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Thomas Gleixner , "Paul E . McKenney" , Boqun Feng , "H . Peter Anvin" , Paul Turner , linux-api@vger.kernel.org, Christian Brauner , Florian Weimer , David.Laight@ACULAB.COM, carlos@redhat.com, Peter Oskolkov , Mathieu Desnoyers Subject: [PATCH v3 02/23] rseq: Introduce extensible rseq ABI Date: Fri, 29 Jul 2022 15:02:04 -0400 Message-Id: <20220729190225.12726-3-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> References: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Introduce the extensible rseq ABI, where the feature size supported by the kernel and the required alignment are communicated to user-space through ELF auxiliary vectors. This allows user-space to call rseq registration with a rseq_len of either 32 bytes for the original struct rseq size (which includes padding), or larger. If rseq_len is larger than 32 bytes, then it must be large enough to contain the feature size communicated to user-space through ELF auxiliary vectors. Signed-off-by: Mathieu Desnoyers --- include/linux/sched.h | 4 ++++ kernel/ptrace.c | 2 +- kernel/rseq.c | 33 +++++++++++++++++++++++++++------ 3 files changed, 32 insertions(+), 7 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index a8911b1f35aa..68b23937b4a5 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1290,6 +1290,7 @@ struct task_struct { =20 #ifdef CONFIG_RSEQ struct rseq __user *rseq; + u32 rseq_len; u32 rseq_sig; /* * RmW on rseq_event_mask must be performed atomically @@ -2282,10 +2283,12 @@ static inline void rseq_fork(struct task_struct *t,= unsigned long clone_flags) { if (clone_flags & CLONE_VM) { t->rseq =3D NULL; + t->rseq_len =3D 0; t->rseq_sig =3D 0; t->rseq_event_mask =3D 0; } else { t->rseq =3D current->rseq; + t->rseq_len =3D current->rseq_len; t->rseq_sig =3D current->rseq_sig; t->rseq_event_mask =3D current->rseq_event_mask; } @@ -2294,6 +2297,7 @@ static inline void rseq_fork(struct task_struct *t, u= nsigned long clone_flags) static inline void rseq_execve(struct task_struct *t) { t->rseq =3D NULL; + t->rseq_len =3D 0; t->rseq_sig =3D 0; t->rseq_event_mask =3D 0; } diff --git a/kernel/ptrace.c b/kernel/ptrace.c index 6149ca5e0e14..390c71e9e573 100644 --- a/kernel/ptrace.c +++ b/kernel/ptrace.c @@ -817,7 +817,7 @@ static long ptrace_get_rseq_configuration(struct task_s= truct *task, { struct ptrace_rseq_configuration conf =3D { .rseq_abi_pointer =3D (u64)(uintptr_t)task->rseq, - .rseq_abi_size =3D sizeof(*task->rseq), + .rseq_abi_size =3D task->rseq_len, .signature =3D task->rseq_sig, .flags =3D 0, }; diff --git a/kernel/rseq.c b/kernel/rseq.c index 97ac20b4f738..46dc5c2ce2b7 100644 --- a/kernel/rseq.c +++ b/kernel/rseq.c @@ -18,6 +18,9 @@ #define CREATE_TRACE_POINTS #include =20 +/* The original rseq structure size (including padding) is 32 bytes. */ +#define ORIG_RSEQ_SIZE 32 + #define RSEQ_CS_PREEMPT_MIGRATE_FLAGS (RSEQ_CS_FLAG_NO_RESTART_ON_MIGRATE = | \ RSEQ_CS_FLAG_NO_RESTART_ON_PREEMPT) =20 @@ -86,10 +89,15 @@ static int rseq_update_cpu_id(struct task_struct *t) u32 cpu_id =3D raw_smp_processor_id(); struct rseq __user *rseq =3D t->rseq; =20 - if (!user_write_access_begin(rseq, sizeof(*rseq))) + if (!user_write_access_begin(rseq, t->rseq_len)) goto efault; unsafe_put_user(cpu_id, &rseq->cpu_id_start, efault_end); unsafe_put_user(cpu_id, &rseq->cpu_id, efault_end); + /* + * Additional feature fields added after ORIG_RSEQ_SIZE + * need to be conditionally updated only if + * t->rseq_len !=3D ORIG_RSEQ_SIZE. + */ user_write_access_end(); trace_rseq_update(t); return 0; @@ -116,6 +124,11 @@ static int rseq_reset_rseq_cpu_id(struct task_struct *= t) */ if (put_user(cpu_id, &t->rseq->cpu_id)) return -EFAULT; + /* + * Additional feature fields added after ORIG_RSEQ_SIZE + * need to be conditionally reset only if + * t->rseq_len !=3D ORIG_RSEQ_SIZE. + */ return 0; } =20 @@ -336,7 +349,7 @@ SYSCALL_DEFINE4(rseq, struct rseq __user *, rseq, u32, = rseq_len, /* Unregister rseq for current thread. */ if (current->rseq !=3D rseq || !current->rseq) return -EINVAL; - if (rseq_len !=3D sizeof(*rseq)) + if (rseq_len !=3D current->rseq_len) return -EINVAL; if (current->rseq_sig !=3D sig) return -EPERM; @@ -345,6 +358,7 @@ SYSCALL_DEFINE4(rseq, struct rseq __user *, rseq, u32, = rseq_len, return ret; current->rseq =3D NULL; current->rseq_sig =3D 0; + current->rseq_len =3D 0; return 0; } =20 @@ -357,7 +371,7 @@ SYSCALL_DEFINE4(rseq, struct rseq __user *, rseq, u32, = rseq_len, * the provided address differs from the prior * one. */ - if (current->rseq !=3D rseq || rseq_len !=3D sizeof(*rseq)) + if (current->rseq !=3D rseq || rseq_len !=3D current->rseq_len) return -EINVAL; if (current->rseq_sig !=3D sig) return -EPERM; @@ -366,15 +380,22 @@ SYSCALL_DEFINE4(rseq, struct rseq __user *, rseq, u32= , rseq_len, } =20 /* - * If there was no rseq previously registered, - * ensure the provided rseq is properly aligned and valid. + * If there was no rseq previously registered, ensure the provided rseq + * is properly aligned, as communcated to user-space through the ELF + * auxiliary vector AT_RSEQ_ALIGN. + * + * In order to be valid, rseq_len is either the original rseq size, or + * large enough to contain all supported fields, as communicated to + * user-space through the ELF auxiliary vector AT_RSEQ_FEATURE_SIZE. */ if (!IS_ALIGNED((unsigned long)rseq, __alignof__(*rseq)) || - rseq_len !=3D sizeof(*rseq)) + rseq_len < ORIG_RSEQ_SIZE || + (rseq_len !=3D ORIG_RSEQ_SIZE && rseq_len < offsetof(struct rseq, end= ))) return -EINVAL; if (!access_ok(rseq, rseq_len)) return -EFAULT; current->rseq =3D rseq; + current->rseq_len =3D rseq_len; current->rseq_sig =3D sig; /* * If rseq was previously inactive, and has just been --=20 2.17.1 From nobody Tue Apr 14 22:46:25 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04BCFC00144 for ; Fri, 29 Jul 2022 19:02:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232587AbiG2TCs (ORCPT ); Fri, 29 Jul 2022 15:02:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50872 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238407AbiG2TCo (ORCPT ); Fri, 29 Jul 2022 15:02:44 -0400 Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E88C988741; Fri, 29 Jul 2022 12:02:42 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 384D1318E8D; Fri, 29 Jul 2022 15:02:42 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id JCEWs45x8sPA; Fri, 29 Jul 2022 15:02:41 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id D2829318B4D; Fri, 29 Jul 2022 15:02:41 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com D2829318B4D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1659121361; bh=NkCVjM+S9iEg/gRamTSXlL2ZBMGnYHhG/cpmn7pzdsM=; h=From:To:Date:Message-Id; b=vHXjUpf6ynyLVX2tJKxWcMZlrEwd080p2Xx9F4MGq2ZpFmgW7EKpmVCrOFD0Ol4bQ i6AM80OogfzmsZ7H2i2syre9K6envqnIoTr/Ezb5nNlQLx5/DUi2piYO1Y+atjkehQ 9kw+frBN9PCw7W2iFr7LBBtzuBQmwfVOyUZehj5gyNZXwC4Atm961wzsRLNPXGYgil ihscdvLpYc/ei5t0ojStR4vvDxSq/6hovty8ftV1MQlSrVr+FtAF2dlO7JolTg+qUq 60w++DbclkJxCiFBqnovxx3PZPkA56Sw22YedJFw3ZLQnJhWjOuNoTLEg4yK7VikXT 86HZjXRUzlH0A== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id oAahqyNA6sEZ; Fri, 29 Jul 2022 15:02:41 -0400 (EDT) Received: from localhost.localdomain (192-222-180-24.qc.cable.ebox.net [192.222.180.24]) by mail.efficios.com (Postfix) with ESMTPSA id 8BCCC3187FF; Fri, 29 Jul 2022 15:02:41 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Thomas Gleixner , "Paul E . McKenney" , Boqun Feng , "H . Peter Anvin" , Paul Turner , linux-api@vger.kernel.org, Christian Brauner , Florian Weimer , David.Laight@ACULAB.COM, carlos@redhat.com, Peter Oskolkov , Mathieu Desnoyers Subject: [PATCH v3 03/23] rseq: extend struct rseq with numa node id Date: Fri, 29 Jul 2022 15:02:05 -0400 Message-Id: <20220729190225.12726-4-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> References: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Adding the NUMA node id to struct rseq is a straightforward thing to do, and a good way to figure out if anything in the user-space ecosystem prevents extending struct rseq. This NUMA node id field allows memory allocators such as tcmalloc to take advantage of fast access to the current NUMA node id to perform NUMA-aware memory allocation. It can also be useful for implementing fast-paths for NUMA-aware user-space mutexes. It also allows implementing getcpu(2) purely in user-space. Signed-off-by: Mathieu Desnoyers --- include/trace/events/rseq.h | 4 +++- include/uapi/linux/rseq.h | 8 ++++++++ kernel/rseq.c | 19 +++++++++++++------ 3 files changed, 24 insertions(+), 7 deletions(-) diff --git a/include/trace/events/rseq.h b/include/trace/events/rseq.h index a04a64bc1a00..6bd442697354 100644 --- a/include/trace/events/rseq.h +++ b/include/trace/events/rseq.h @@ -16,13 +16,15 @@ TRACE_EVENT(rseq_update, =20 TP_STRUCT__entry( __field(s32, cpu_id) + __field(s32, node_id) ), =20 TP_fast_assign( __entry->cpu_id =3D raw_smp_processor_id(); + __entry->node_id =3D cpu_to_node(raw_smp_processor_id()); ), =20 - TP_printk("cpu_id=3D%d", __entry->cpu_id) + TP_printk("cpu_id=3D%d node_id=3D%d", __entry->cpu_id, __entry->node_id) ); =20 TRACE_EVENT(rseq_ip_fixup, diff --git a/include/uapi/linux/rseq.h b/include/uapi/linux/rseq.h index 05d3c4cdeb40..1cb90a435c5c 100644 --- a/include/uapi/linux/rseq.h +++ b/include/uapi/linux/rseq.h @@ -131,6 +131,14 @@ struct rseq { */ __u32 flags; =20 + /* + * Restartable sequences node_id field. Updated by the kernel. Read by + * user-space with single-copy atomicity semantics. This field should + * only be read by the thread which registered this data structure. + * Aligned on 32-bit. Contains the current NUMA node ID. + */ + __u32 node_id; + /* * Flexible array member at end of structure, after last feature field. */ diff --git a/kernel/rseq.c b/kernel/rseq.c index 46dc5c2ce2b7..cb7d8a5afc82 100644 --- a/kernel/rseq.c +++ b/kernel/rseq.c @@ -84,15 +84,17 @@ * F1. */ =20 -static int rseq_update_cpu_id(struct task_struct *t) +static int rseq_update_cpu_node_id(struct task_struct *t) { - u32 cpu_id =3D raw_smp_processor_id(); struct rseq __user *rseq =3D t->rseq; + u32 cpu_id =3D raw_smp_processor_id(); + u32 node_id =3D cpu_to_node(cpu_id); =20 if (!user_write_access_begin(rseq, t->rseq_len)) goto efault; unsafe_put_user(cpu_id, &rseq->cpu_id_start, efault_end); unsafe_put_user(cpu_id, &rseq->cpu_id, efault_end); + unsafe_put_user(node_id, &rseq->node_id, efault_end); /* * Additional feature fields added after ORIG_RSEQ_SIZE * need to be conditionally updated only if @@ -108,9 +110,9 @@ static int rseq_update_cpu_id(struct task_struct *t) return -EFAULT; } =20 -static int rseq_reset_rseq_cpu_id(struct task_struct *t) +static int rseq_reset_rseq_cpu_node_id(struct task_struct *t) { - u32 cpu_id_start =3D 0, cpu_id =3D RSEQ_CPU_ID_UNINITIALIZED; + u32 cpu_id_start =3D 0, cpu_id =3D RSEQ_CPU_ID_UNINITIALIZED, node_id =3D= 0; =20 /* * Reset cpu_id_start to its initial state (0). @@ -124,6 +126,11 @@ static int rseq_reset_rseq_cpu_id(struct task_struct *= t) */ if (put_user(cpu_id, &t->rseq->cpu_id)) return -EFAULT; + /* + * Reset node_id to its initial state (0). + */ + if (put_user(node_id, &t->rseq->node_id)) + return -EFAULT; /* * Additional feature fields added after ORIG_RSEQ_SIZE * need to be conditionally reset only if @@ -306,7 +313,7 @@ void __rseq_handle_notify_resume(struct ksignal *ksig, = struct pt_regs *regs) if (unlikely(ret < 0)) goto error; } - if (unlikely(rseq_update_cpu_id(t))) + if (unlikely(rseq_update_cpu_node_id(t))) goto error; return; =20 @@ -353,7 +360,7 @@ SYSCALL_DEFINE4(rseq, struct rseq __user *, rseq, u32, = rseq_len, return -EINVAL; if (current->rseq_sig !=3D sig) return -EPERM; - ret =3D rseq_reset_rseq_cpu_id(current); + ret =3D rseq_reset_rseq_cpu_node_id(current); if (ret) return ret; current->rseq =3D NULL; --=20 2.17.1 From nobody Tue Apr 14 22:46:25 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0335EC19F2C for ; Fri, 29 Jul 2022 19:03:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234214AbiG2TDB (ORCPT ); Fri, 29 Jul 2022 15:03:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233409AbiG2TCp (ORCPT ); Fri, 29 Jul 2022 15:02:45 -0400 Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6D79288CED; Fri, 29 Jul 2022 12:02:43 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 9A22B318A68; Fri, 29 Jul 2022 15:02:42 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 4MQv8Fyh4pWs; Fri, 29 Jul 2022 15:02:42 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 2B974318C6A; Fri, 29 Jul 2022 15:02:42 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 2B974318C6A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1659121362; bh=Yg75Hv5F0s9zyJSBB4UegGnFfDCNyw8vYEQfDAATINI=; h=From:To:Date:Message-Id; b=LzXSJBWE221CDApTBy1wZ1TUdsKROkvDfmtyke/ZV1K//zDPd624i9n4OZta3Q1SP MWC+JL/6yj8pQPiNgN7KADEPXUc0ka+YQryH+p8Jb4UBkaWZlkNY6dWwIewjr3NNIR +bhH1QMfX3FKiJjUDbtxMKlo+n5J4HU+Ork1+xL67zh2x+EQ0sYnDjqFWn2CrVmS4k qbOAFxvexA875BB5CNdHkSaEnBef7f8DhP1s1ZfpNWyiTG0TAxA1LnDMA4a5cIj0U+ SU8J2XUbke6IZ0TDWuDuIiRmrJKAy6kuWEMH+sF0ADsLDAYQex5Ciwlo9BCGql5+h9 mC+3bJ2PFhKmw== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id eyiUjJlVELEV; Fri, 29 Jul 2022 15:02:42 -0400 (EDT) Received: from localhost.localdomain (192-222-180-24.qc.cable.ebox.net [192.222.180.24]) by mail.efficios.com (Postfix) with ESMTPSA id D2F57318F09; Fri, 29 Jul 2022 15:02:41 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Thomas Gleixner , "Paul E . McKenney" , Boqun Feng , "H . Peter Anvin" , Paul Turner , linux-api@vger.kernel.org, Christian Brauner , Florian Weimer , David.Laight@ACULAB.COM, carlos@redhat.com, Peter Oskolkov , Mathieu Desnoyers Subject: [PATCH v3 04/23] selftests/rseq: Use ELF auxiliary vector for extensible rseq Date: Fri, 29 Jul 2022 15:02:06 -0400 Message-Id: <20220729190225.12726-5-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> References: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Use the ELF auxiliary vector AT_RSEQ_FEATURE_SIZE to detect the RSEQ features supported by the kernel. Signed-off-by: Mathieu Desnoyers --- tools/testing/selftests/rseq/rseq-abi.h | 5 ++ tools/testing/selftests/rseq/rseq.c | 68 ++++++++++++++++++++++--- tools/testing/selftests/rseq/rseq.h | 18 +++++-- 3 files changed, 79 insertions(+), 12 deletions(-) diff --git a/tools/testing/selftests/rseq/rseq-abi.h b/tools/testing/selfte= sts/rseq/rseq-abi.h index a8c44d9af71f..00ac846d85b0 100644 --- a/tools/testing/selftests/rseq/rseq-abi.h +++ b/tools/testing/selftests/rseq/rseq-abi.h @@ -146,6 +146,11 @@ struct rseq_abi { * this thread. */ __u32 flags; + + /* + * Flexible array member at end of structure, after last feature field. + */ + char end[]; } __attribute__((aligned(4 * sizeof(__u64)))); =20 #endif /* _RSEQ_ABI_H */ diff --git a/tools/testing/selftests/rseq/rseq.c b/tools/testing/selftests/= rseq/rseq.c index 986b9458efb2..20ea536d1012 100644 --- a/tools/testing/selftests/rseq/rseq.c +++ b/tools/testing/selftests/rseq/rseq.c @@ -28,6 +28,8 @@ #include #include #include +#include +#include =20 #include "../kselftest.h" #include "rseq.h" @@ -36,20 +38,38 @@ static const ptrdiff_t *libc_rseq_offset_p; static const unsigned int *libc_rseq_size_p; static const unsigned int *libc_rseq_flags_p; =20 -/* Offset from the thread pointer to the rseq area. */ +/* Offset from the thread pointer to the rseq area. */ ptrdiff_t rseq_offset; =20 -/* Size of the registered rseq area. 0 if the registration was - unsuccessful. */ +/* + * Size of the registered rseq area. 0 if the registration was + * unsuccessful. + */ unsigned int rseq_size =3D -1U; =20 /* Flags used during rseq registration. */ unsigned int rseq_flags; =20 +/* + * rseq feature size supported by the kernel. 0 if the registration was + * unsuccessful. + */ +unsigned int rseq_feature_size =3D -1U; + static int rseq_ownership; +static int rseq_reg_success; /* At least one rseq registration has succede= d. */ + +/* Allocate a large area for the TLS. */ +#define RSEQ_THREAD_AREA_ALLOC_SIZE 1024 + +/* Original struct rseq feature size is 20 bytes. */ +#define ORIG_RSEQ_FEATURE_SIZE 20 + +/* Orignal struct rseq allocation size is 32 bytes. */ +#define ORIG_RSEQ_ALLOC_SIZE 32 =20 static -__thread struct rseq_abi __rseq_abi __attribute__((tls_model("initial-exec= "))) =3D { +__thread struct rseq_abi __rseq_abi __attribute__((tls_model("initial-exec= "), aligned(RSEQ_THREAD_AREA_ALLOC_SIZE))) =3D { .cpu_id =3D RSEQ_ABI_CPU_ID_UNINITIALIZED, }; =20 @@ -84,10 +104,18 @@ int rseq_register_current_thread(void) /* Treat libc's ownership as a successful registration. */ return 0; } - rc =3D sys_rseq(&__rseq_abi, sizeof(struct rseq_abi), 0, RSEQ_SIG); - if (rc) + rc =3D sys_rseq(&__rseq_abi, rseq_size, 0, RSEQ_SIG); + if (rc) { + if (RSEQ_READ_ONCE(rseq_reg_success)) { + /* Incoherent success/failure within process. */ + abort(); + } + rseq_size =3D 0; + rseq_feature_size =3D 0; return -1; + } assert(rseq_current_cpu_raw() >=3D 0); + RSEQ_WRITE_ONCE(rseq_reg_success, 1); return 0; } =20 @@ -99,12 +127,28 @@ int rseq_unregister_current_thread(void) /* Treat libc's ownership as a successful unregistration. */ return 0; } - rc =3D sys_rseq(&__rseq_abi, sizeof(struct rseq_abi), RSEQ_ABI_FLAG_UNREG= ISTER, RSEQ_SIG); + rc =3D sys_rseq(&__rseq_abi, rseq_size, RSEQ_ABI_FLAG_UNREGISTER, RSEQ_SI= G); if (rc) return -1; return 0; } =20 +static +unsigned int get_rseq_feature_size(void) +{ + unsigned long auxv_rseq_feature_size, auxv_rseq_align; + + auxv_rseq_align =3D getauxval(AT_RSEQ_ALIGN); + assert(!auxv_rseq_align || auxv_rseq_align <=3D RSEQ_THREAD_AREA_ALLOC_SI= ZE); + + auxv_rseq_feature_size =3D getauxval(AT_RSEQ_FEATURE_SIZE); + assert(!auxv_rseq_feature_size || auxv_rseq_feature_size <=3D RSEQ_THREAD= _AREA_ALLOC_SIZE); + if (auxv_rseq_feature_size) + return auxv_rseq_feature_size; + else + return ORIG_RSEQ_FEATURE_SIZE; +} + static __attribute__((constructor)) void rseq_init(void) { @@ -116,14 +160,21 @@ void rseq_init(void) rseq_offset =3D *libc_rseq_offset_p; rseq_size =3D *libc_rseq_size_p; rseq_flags =3D *libc_rseq_flags_p; + rseq_feature_size =3D get_rseq_feature_size(); + if (rseq_feature_size > rseq_size) + rseq_feature_size =3D rseq_size; return; } if (!rseq_available()) return; rseq_ownership =3D 1; rseq_offset =3D (void *)&__rseq_abi - rseq_thread_pointer(); - rseq_size =3D sizeof(struct rseq_abi); rseq_flags =3D 0; + rseq_feature_size =3D get_rseq_feature_size(); + if (rseq_feature_size =3D=3D ORIG_RSEQ_FEATURE_SIZE) + rseq_size =3D ORIG_RSEQ_ALLOC_SIZE; + else + rseq_size =3D RSEQ_THREAD_AREA_ALLOC_SIZE; } =20 static __attribute__((destructor)) @@ -133,6 +184,7 @@ void rseq_exit(void) return; rseq_offset =3D 0; rseq_size =3D -1U; + rseq_feature_size =3D -1U; rseq_ownership =3D 0; } =20 diff --git a/tools/testing/selftests/rseq/rseq.h b/tools/testing/selftests/= rseq/rseq.h index 6f7513384bf5..95adc1e1b0db 100644 --- a/tools/testing/selftests/rseq/rseq.h +++ b/tools/testing/selftests/rseq/rseq.h @@ -47,14 +47,24 @@ =20 #include "rseq-thread-pointer.h" =20 -/* Offset from the thread pointer to the rseq area. */ +/* Offset from the thread pointer to the rseq area. */ extern ptrdiff_t rseq_offset; -/* Size of the registered rseq area. 0 if the registration was - unsuccessful. */ + +/* + * Size of the registered rseq area. 0 if the registration was + * unsuccessful. + */ extern unsigned int rseq_size; -/* Flags used during rseq registration. */ + +/* Flags used during rseq registration. */ extern unsigned int rseq_flags; =20 +/* + * rseq feature size supported by the kernel. 0 if the registration was + * unsuccessful. + */ +extern unsigned int rseq_feature_size; + static inline struct rseq_abi *rseq_get_abi(void) { return (struct rseq_abi *) ((uintptr_t) rseq_thread_pointer() + rseq_offs= et); --=20 2.17.1 From nobody Tue Apr 14 22:46:25 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 822C2C00144 for ; Fri, 29 Jul 2022 19:03:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238656AbiG2TDG (ORCPT ); Fri, 29 Jul 2022 15:03:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50908 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238461AbiG2TCp (ORCPT ); Fri, 29 Jul 2022 15:02:45 -0400 Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 20B0B66108; Fri, 29 Jul 2022 12:02:45 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id DF657318F0D; Fri, 29 Jul 2022 15:02:42 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id QJSUBYdRK_jN; Fri, 29 Jul 2022 15:02:42 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 78E2F318D75; Fri, 29 Jul 2022 15:02:42 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 78E2F318D75 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1659121362; bh=O6oX4tqm54q4o/zImsHHSCyS1HipqEmhYnQTSLOU7vM=; h=From:To:Date:Message-Id; b=WBSo43I3fzFvtiPNfSZ1nVfarX3dKD4vdOSdIUGFyMERoLUn++xdwZxbtq63V2263 yljpX9YFlwBPbNm0vpEl6//kUmdsxCWVRV2uDGJdi6QPLgMZ92xyV2umKDGQrZ4AFh XyU/yfT0SpSJTdQb75OyBg6OGhXW75bDK8z5SA4rySvHWNTGwJxn1ABxLyqcv8d+CP EshzOj1ioAB/BrQqH815a0wuInAcn6wFN+JeNO/cpzGZS7s7+kJehmFBB3R7dIINZZ X27f/tncRH2aadhQJTzdSPDPgsQeTHIBKoppMdAcWOpwr+AFnBrNfP8Zf5x7dartvE b/A12N/X60Cig== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id t1Mq1S7JASDh; Fri, 29 Jul 2022 15:02:42 -0400 (EDT) Received: from localhost.localdomain (192-222-180-24.qc.cable.ebox.net [192.222.180.24]) by mail.efficios.com (Postfix) with ESMTPSA id 2E877318DBD; Fri, 29 Jul 2022 15:02:42 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Thomas Gleixner , "Paul E . McKenney" , Boqun Feng , "H . Peter Anvin" , Paul Turner , linux-api@vger.kernel.org, Christian Brauner , Florian Weimer , David.Laight@ACULAB.COM, carlos@redhat.com, Peter Oskolkov , Mathieu Desnoyers Subject: [PATCH v3 05/23] selftests/rseq: Implement rseq numa node id field selftest Date: Fri, 29 Jul 2022 15:02:07 -0400 Message-Id: <20220729190225.12726-6-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> References: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Test the NUMA node id extension rseq field. Compare it against the value returned by the getcpu(2) system call while pinned on a specific core. Signed-off-by: Mathieu Desnoyers --- tools/testing/selftests/rseq/basic_test.c | 4 ++++ tools/testing/selftests/rseq/rseq-abi.h | 8 +++++++ tools/testing/selftests/rseq/rseq.c | 18 +++++++++++++++ tools/testing/selftests/rseq/rseq.h | 28 +++++++++++++++++++++++ 4 files changed, 58 insertions(+) diff --git a/tools/testing/selftests/rseq/basic_test.c b/tools/testing/self= tests/rseq/basic_test.c index d8efbfb89193..295eea16466f 100644 --- a/tools/testing/selftests/rseq/basic_test.c +++ b/tools/testing/selftests/rseq/basic_test.c @@ -22,6 +22,8 @@ void test_cpu_pointer(void) CPU_ZERO(&test_affinity); for (i =3D 0; i < CPU_SETSIZE; i++) { if (CPU_ISSET(i, &affinity)) { + int node; + CPU_SET(i, &test_affinity); sched_setaffinity(0, sizeof(test_affinity), &test_affinity); @@ -29,6 +31,8 @@ void test_cpu_pointer(void) assert(rseq_current_cpu() =3D=3D i); assert(rseq_current_cpu_raw() =3D=3D i); assert(rseq_cpu_start() =3D=3D i); + node =3D rseq_fallback_current_node(); + assert(rseq_current_node_id() =3D=3D node); CPU_CLR(i, &test_affinity); } } diff --git a/tools/testing/selftests/rseq/rseq-abi.h b/tools/testing/selfte= sts/rseq/rseq-abi.h index 00ac846d85b0..a1faa9162d52 100644 --- a/tools/testing/selftests/rseq/rseq-abi.h +++ b/tools/testing/selftests/rseq/rseq-abi.h @@ -147,6 +147,14 @@ struct rseq_abi { */ __u32 flags; =20 + /* + * Restartable sequences node_id field. Updated by the kernel. Read by + * user-space with single-copy atomicity semantics. This field should + * only be read by the thread which registered this data structure. + * Aligned on 32-bit. Contains the current NUMA node ID. + */ + __u32 node_id; + /* * Flexible array member at end of structure, after last feature field. */ diff --git a/tools/testing/selftests/rseq/rseq.c b/tools/testing/selftests/= rseq/rseq.c index 20ea536d1012..0a96c3c779cd 100644 --- a/tools/testing/selftests/rseq/rseq.c +++ b/tools/testing/selftests/rseq/rseq.c @@ -79,6 +79,11 @@ static int sys_rseq(struct rseq_abi *rseq_abi, uint32_t = rseq_len, return syscall(__NR_rseq, rseq_abi, rseq_len, flags, sig); } =20 +static int sys_getcpu(unsigned *cpu, unsigned *node) +{ + return syscall(__NR_getcpu, cpu, node, NULL); +} + int rseq_available(void) { int rc; @@ -199,3 +204,16 @@ int32_t rseq_fallback_current_cpu(void) } return cpu; } + +int32_t rseq_fallback_current_node(void) +{ + uint32_t cpu_id, node_id; + int ret; + + ret =3D sys_getcpu(&cpu_id, &node_id); + if (ret) { + perror("sys_getcpu()"); + return ret; + } + return (int32_t) node_id; +} diff --git a/tools/testing/selftests/rseq/rseq.h b/tools/testing/selftests/= rseq/rseq.h index 95adc1e1b0db..fd17d0e54a1b 100644 --- a/tools/testing/selftests/rseq/rseq.h +++ b/tools/testing/selftests/rseq/rseq.h @@ -20,6 +20,15 @@ #include "rseq-abi.h" #include "compiler.h" =20 +#ifndef rseq_sizeof_field +#define rseq_sizeof_field(TYPE, MEMBER) sizeof((((TYPE *)0)->MEMBER)) +#endif + +#ifndef rseq_offsetofend +#define rseq_offsetofend(TYPE, MEMBER) \ + (offsetof(TYPE, MEMBER) + rseq_sizeof_field(TYPE, MEMBER)) +#endif + /* * Empty code injection macros, override when testing. * It is important to consider that the ASM injection macros need to be @@ -128,6 +137,11 @@ int rseq_unregister_current_thread(void); */ int32_t rseq_fallback_current_cpu(void); =20 +/* + * Restartable sequence fallback for reading the current node number. + */ +int32_t rseq_fallback_current_node(void); + /* * Values returned can be either the current CPU number, -1 (rseq is * uninitialized), or -2 (rseq initialization has failed). @@ -163,6 +177,20 @@ static inline uint32_t rseq_current_cpu(void) return cpu; } =20 +static inline bool rseq_node_id_available(void) +{ + return (int) rseq_feature_size >=3D rseq_offsetofend(struct rseq_abi, nod= e_id); +} + +/* + * Current NUMA node number. + */ +static inline uint32_t rseq_current_node_id(void) +{ + assert(rseq_node_id_available()); + return RSEQ_ACCESS_ONCE(rseq_get_abi()->node_id); +} + static inline void rseq_clear_rseq_cs(void) { RSEQ_WRITE_ONCE(rseq_get_abi()->rseq_cs.arch.ptr, 0); --=20 2.17.1 From nobody Tue Apr 14 22:46:25 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDF92C00144 for ; Fri, 29 Jul 2022 19:03:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238723AbiG2TDS (ORCPT ); Fri, 29 Jul 2022 15:03:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51112 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238577AbiG2TCy (ORCPT ); Fri, 29 Jul 2022 15:02:54 -0400 Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4C18D88E0B; Fri, 29 Jul 2022 12:02:49 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 3847B318DCB; Fri, 29 Jul 2022 15:02:48 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id WCN-K7vRmKAZ; Fri, 29 Jul 2022 15:02:47 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 97639318DC4; Fri, 29 Jul 2022 15:02:44 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 97639318DC4 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1659121364; bh=H1BI137TQVD0IUIV2Akwp3DX/CDXZ2SZLX9a456H6tM=; h=From:To:Date:Message-Id; b=uVb6XYNWUDjIIcjcFprb8OWL25iatpTSi/XbLIGX4C0DJnmzzvok/710No7TnmIA5 IcCNESSSsv2ESuvBDAp2Gpw3sfVI/e7v5N3sNAvJ3fqlP3rQ9wlm6dFy729YI8Y40X 8JLTRqzoy7lAHNcs4doL6WTFJ3el/UnDc9K49WkC29N+OYZLTw2ih6NG1cgxp3jrR8 XNfSweydw6XTzz678vt4BLDT+TNRqnSUiXF9ksnmjM3EqyEEkMK3guimlBMcP9a+b4 gRUwIn1ivvymg9T2rUbiTJe4O6HJKLqcUjnjb1uG4qJ+IlH2rdnrYlXjgoh9uJ+Vwt FlHTAa85MaRXg== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 8xRFzbItOyLY; Fri, 29 Jul 2022 15:02:44 -0400 (EDT) Received: from localhost.localdomain (192-222-180-24.qc.cable.ebox.net [192.222.180.24]) by mail.efficios.com (Postfix) with ESMTPSA id 7A370318F0C; Fri, 29 Jul 2022 15:02:42 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Thomas Gleixner , "Paul E . McKenney" , Boqun Feng , "H . Peter Anvin" , Paul Turner , linux-api@vger.kernel.org, Christian Brauner , Florian Weimer , David.Laight@ACULAB.COM, carlos@redhat.com, Peter Oskolkov , Mathieu Desnoyers Subject: [PATCH v3 06/23] lib: invert _find_next_bit source arguments Date: Fri, 29 Jul 2022 15:02:08 -0400 Message-Id: <20220729190225.12726-7-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> References: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Apply bit-invert operations before the AND operation in _find_next_bit. Allows AND operations on combined bitmasks in which we search either for one or zero, e.g.: find first bit which is both zero in one bitmask AND one in the second bitmask. The existing use for find first zero bit does not use the second argument, so whether the inversion is performed before or after the AND operator does not matter. Signed-off-by: Mathieu Desnoyers --- include/linux/find.h | 13 +++++++------ lib/find_bit.c | 17 ++++++++--------- tools/include/linux/find.h | 9 +++++---- tools/lib/find_bit.c | 17 ++++++++--------- 4 files changed, 28 insertions(+), 28 deletions(-) diff --git a/include/linux/find.h b/include/linux/find.h index 5bb6db213bcb..41941cb9cad7 100644 --- a/include/linux/find.h +++ b/include/linux/find.h @@ -10,7 +10,8 @@ =20 extern unsigned long _find_next_bit(const unsigned long *addr1, const unsigned long *addr2, unsigned long nbits, - unsigned long start, unsigned long invert, unsigned long le); + unsigned long start, unsigned long invert_src1, + unsigned long src2, unsigned long le); extern unsigned long _find_first_bit(const unsigned long *addr, unsigned l= ong size); extern unsigned long _find_first_and_bit(const unsigned long *addr1, const unsigned long *addr2, unsigned long size); @@ -41,7 +42,7 @@ unsigned long find_next_bit(const unsigned long *addr, un= signed long size, return val ? __ffs(val) : size; } =20 - return _find_next_bit(addr, NULL, size, offset, 0UL, 0); + return _find_next_bit(addr, NULL, size, offset, 0UL, 0UL, 0); } #endif =20 @@ -71,7 +72,7 @@ unsigned long find_next_and_bit(const unsigned long *addr= 1, return val ? __ffs(val) : size; } =20 - return _find_next_bit(addr1, addr2, size, offset, 0UL, 0); + return _find_next_bit(addr1, addr2, size, offset, 0UL, 0UL, 0); } #endif =20 @@ -99,7 +100,7 @@ unsigned long find_next_zero_bit(const unsigned long *ad= dr, unsigned long size, return val =3D=3D ~0UL ? size : ffz(val); } =20 - return _find_next_bit(addr, NULL, size, offset, ~0UL, 0); + return _find_next_bit(addr, NULL, size, offset, ~0UL, 0UL, 0); } #endif =20 @@ -247,7 +248,7 @@ unsigned long find_next_zero_bit_le(const void *addr, u= nsigned return val =3D=3D ~0UL ? size : ffz(val); } =20 - return _find_next_bit(addr, NULL, size, offset, ~0UL, 1); + return _find_next_bit(addr, NULL, size, offset, ~0UL, 0UL, 1); } #endif =20 @@ -266,7 +267,7 @@ unsigned long find_next_bit_le(const void *addr, unsign= ed return val ? __ffs(val) : size; } =20 - return _find_next_bit(addr, NULL, size, offset, 0UL, 1); + return _find_next_bit(addr, NULL, size, offset, 0UL, 0UL, 1); } #endif =20 diff --git a/lib/find_bit.c b/lib/find_bit.c index 1b8e4b2a9cba..73e78565e691 100644 --- a/lib/find_bit.c +++ b/lib/find_bit.c @@ -25,23 +25,23 @@ /* * This is a common helper function for find_next_bit, find_next_zero_bit,= and * find_next_and_bit. The differences are: - * - The "invert" argument, which is XORed with each fetched word before - * searching it for one bits. + * - The "invert_src1" and "invert_src2" arguments, which are XORed to + * each source word before applying the 'and' operator. * - The optional "addr2", which is anded with "addr1" if present. */ unsigned long _find_next_bit(const unsigned long *addr1, const unsigned long *addr2, unsigned long nbits, - unsigned long start, unsigned long invert, unsigned long le) + unsigned long start, unsigned long invert_src1, + unsigned long invert_src2, unsigned long le) { unsigned long tmp, mask; =20 if (unlikely(start >=3D nbits)) return nbits; =20 - tmp =3D addr1[start / BITS_PER_LONG]; + tmp =3D addr1[start / BITS_PER_LONG] ^ invert_src1; if (addr2) - tmp &=3D addr2[start / BITS_PER_LONG]; - tmp ^=3D invert; + tmp &=3D addr2[start / BITS_PER_LONG] ^ invert_src2; =20 /* Handle 1st word. */ mask =3D BITMAP_FIRST_WORD_MASK(start); @@ -57,10 +57,9 @@ unsigned long _find_next_bit(const unsigned long *addr1, if (start >=3D nbits) return nbits; =20 - tmp =3D addr1[start / BITS_PER_LONG]; + tmp =3D addr1[start / BITS_PER_LONG] ^ invert_src1; if (addr2) - tmp &=3D addr2[start / BITS_PER_LONG]; - tmp ^=3D invert; + tmp &=3D addr2[start / BITS_PER_LONG] ^ invert_src2; } =20 if (le) diff --git a/tools/include/linux/find.h b/tools/include/linux/find.h index 47e2bd6c5174..5ab0c95086ad 100644 --- a/tools/include/linux/find.h +++ b/tools/include/linux/find.h @@ -10,7 +10,8 @@ =20 extern unsigned long _find_next_bit(const unsigned long *addr1, const unsigned long *addr2, unsigned long nbits, - unsigned long start, unsigned long invert, unsigned long le); + unsigned long start, unsigned long invert_src1, + unsigned long src2, unsigned long le); extern unsigned long _find_first_bit(const unsigned long *addr, unsigned l= ong size); extern unsigned long _find_first_and_bit(const unsigned long *addr1, const unsigned long *addr2, unsigned long size); @@ -41,7 +42,7 @@ unsigned long find_next_bit(const unsigned long *addr, un= signed long size, return val ? __ffs(val) : size; } =20 - return _find_next_bit(addr, NULL, size, offset, 0UL, 0); + return _find_next_bit(addr, NULL, size, offset, 0UL, 0UL, 0); } #endif =20 @@ -71,7 +72,7 @@ unsigned long find_next_and_bit(const unsigned long *addr= 1, return val ? __ffs(val) : size; } =20 - return _find_next_bit(addr1, addr2, size, offset, 0UL, 0); + return _find_next_bit(addr1, addr2, size, offset, 0UL, 0UL, 0); } #endif =20 @@ -99,7 +100,7 @@ unsigned long find_next_zero_bit(const unsigned long *ad= dr, unsigned long size, return val =3D=3D ~0UL ? size : ffz(val); } =20 - return _find_next_bit(addr, NULL, size, offset, ~0UL, 0); + return _find_next_bit(addr, NULL, size, offset, ~0UL, 0UL, 0); } #endif =20 diff --git a/tools/lib/find_bit.c b/tools/lib/find_bit.c index ba4b8d94e004..4176232de7f9 100644 --- a/tools/lib/find_bit.c +++ b/tools/lib/find_bit.c @@ -24,13 +24,14 @@ /* * This is a common helper function for find_next_bit, find_next_zero_bit,= and * find_next_and_bit. The differences are: - * - The "invert" argument, which is XORed with each fetched word before - * searching it for one bits. + * - The "invert_src1" and "invert_src2" arguments, which are XORed to + * each source word before applying the 'and' operator. * - The optional "addr2", which is anded with "addr1" if present. */ unsigned long _find_next_bit(const unsigned long *addr1, const unsigned long *addr2, unsigned long nbits, - unsigned long start, unsigned long invert, unsigned long le) + unsigned long start, unsigned long invert_src1, + unsigned long invert_src2, unsigned long le) { unsigned long tmp, mask; (void) le; @@ -38,10 +39,9 @@ unsigned long _find_next_bit(const unsigned long *addr1, if (unlikely(start >=3D nbits)) return nbits; =20 - tmp =3D addr1[start / BITS_PER_LONG]; + tmp =3D addr1[start / BITS_PER_LONG] ^ invert_src1; if (addr2) - tmp &=3D addr2[start / BITS_PER_LONG]; - tmp ^=3D invert; + tmp &=3D addr2[start / BITS_PER_LONG] ^ invert_src2; =20 /* Handle 1st word. */ mask =3D BITMAP_FIRST_WORD_MASK(start); @@ -64,10 +64,9 @@ unsigned long _find_next_bit(const unsigned long *addr1, if (start >=3D nbits) return nbits; =20 - tmp =3D addr1[start / BITS_PER_LONG]; + tmp =3D addr1[start / BITS_PER_LONG] ^ invert_src1; if (addr2) - tmp &=3D addr2[start / BITS_PER_LONG]; - tmp ^=3D invert; + tmp &=3D addr2[start / BITS_PER_LONG] ^ invert_src2; } =20 #if (0) --=20 2.17.1 From nobody Tue Apr 14 22:46:25 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E94C6C00144 for ; Fri, 29 Jul 2022 19:03:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238703AbiG2TDP (ORCPT ); Fri, 29 Jul 2022 15:03:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50992 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238542AbiG2TCt (ORCPT ); Fri, 29 Jul 2022 15:02:49 -0400 Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C86FC88CED; Fri, 29 Jul 2022 12:02:48 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id A09ED318B57; Fri, 29 Jul 2022 15:02:47 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id ehFCq0xdU8uH; Fri, 29 Jul 2022 15:02:47 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 975F931900F; Fri, 29 Jul 2022 15:02:44 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 975F931900F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1659121364; bh=8X0HKRr4vzicIb3vy2m1DCC4ns3X3lY0CyPw6s0VI/k=; h=From:To:Date:Message-Id; b=agBVs/r8a9Dextj6marfS4RLkIYIzjXzu7uTHbH9zvjmfPIXwWnRos+LaSZ8UEMEI E1pagUe9zljyh08AzrsGimRS5oGMIYNhu2ckwOX8Zm0aG0i1XrcbBtme3CAaUQ1GHu xZQZSJB65WBeQKPuzqGK/kW8A+NVuNirOQLIbKOyP8BTPV+0wuNBzJndwo/QbxirOj L5FRnfAj5O/79DmdN5uv1ojLgSfD3wwtQl1Nazo/mIL39GnlUVuKyYAgTwoupLudfj 9QEq3NCBw+NW9L+1fzLXN37avennCeHTmpjKAEciIMFaXwGxHK9fWlejWGlbpVtG0r thkppT1RRf7Xg== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id h6mpaK2qQ_Jr; Fri, 29 Jul 2022 15:02:44 -0400 (EDT) Received: from localhost.localdomain (192-222-180-24.qc.cable.ebox.net [192.222.180.24]) by mail.efficios.com (Postfix) with ESMTPSA id CA6E9318E8E; Fri, 29 Jul 2022 15:02:42 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Thomas Gleixner , "Paul E . McKenney" , Boqun Feng , "H . Peter Anvin" , Paul Turner , linux-api@vger.kernel.org, Christian Brauner , Florian Weimer , David.Laight@ACULAB.COM, carlos@redhat.com, Peter Oskolkov , Mathieu Desnoyers Subject: [PATCH v3 07/23] lib: implement find_{first,next}_{zero,one}_and_zero_bit Date: Fri, 29 Jul 2022 15:02:09 -0400 Message-Id: <20220729190225.12726-8-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> References: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Allow finding the first or next bit within two input bitmasks which is either: - both zero and zero, - respectively one and zero. Signed-off-by: Mathieu Desnoyers --- include/linux/find.h | 110 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 110 insertions(+) diff --git a/include/linux/find.h b/include/linux/find.h index 41941cb9cad7..ad446f248b36 100644 --- a/include/linux/find.h +++ b/include/linux/find.h @@ -76,6 +76,66 @@ unsigned long find_next_and_bit(const unsigned long *add= r1, } #endif =20 +#ifndef find_next_one_and_zero_bit +/** + * find_next_one_and_zero_bit - find the next bit which is one in addr1 an= d zero in addr2 memory region + * @addr1: The first address to base the search on + * @addr2: The second address to base the search on + * @offset: The bitnumber to start searching at + * @size: The bitmap size in bits + * + * Returns the bit number for the next bit set in addr1 and cleared in add= r2 + * If no corresponding bits meet this criterion, returns @size. + */ +static inline +unsigned long find_next_one_and_zero_bit(const unsigned long *addr1, + const unsigned long *addr2, unsigned long size, + unsigned long offset) +{ + if (small_const_nbits(size)) { + unsigned long val; + + if (unlikely(offset >=3D size)) + return size; + + val =3D *addr1 & ~*addr2 & GENMASK(size - 1, offset); + return val ? __ffs(val) : size; + } + + return _find_next_bit(addr1, addr2, size, offset, 0UL, ~0UL, 0); +} +#endif + +#ifndef find_next_zero_and_zero_bit +/** + * find_next_zero_and_zero_bit - find the next bit which is zero in addr1 = and addr2 memory regions + * @addr1: The first address to base the search on + * @addr2: The second address to base the search on + * @offset: The bitnumber to start searching at + * @size: The bitmap size in bits + * + * Returns the bit number for the next bit cleared in addr1 and addr2 + * If no corresponding bits meet this criterion, returns @size. + */ +static inline +unsigned long find_next_zero_and_zero_bit(const unsigned long *addr1, + const unsigned long *addr2, unsigned long size, + unsigned long offset) +{ + if (small_const_nbits(size)) { + unsigned long val; + + if (unlikely(offset >=3D size)) + return size; + + val =3D ~*addr1 & ~*addr2 & GENMASK(size - 1, offset); + return val ? __ffs(val) : size; + } + + return _find_next_bit(addr1, addr2, size, offset, ~0UL, ~0UL, 0); +} +#endif + #ifndef find_next_zero_bit /** * find_next_zero_bit - find the next cleared bit in a memory region @@ -173,6 +233,56 @@ unsigned long find_first_zero_bit(const unsigned long = *addr, unsigned long size) } #endif =20 +#ifndef find_first_one_and_zero_bit +/** + * find_first_one_and_zero_bit - find the first bit which is one in addr1 = and zero in addr2 memory region + * @addr1: The first address to base the search on + * @addr2: The second address to base the search on + * @size: The bitmap size in bits + * + * Returns the bit number for the first bit set in addr1 and cleared in ad= dr2 + * If no corresponding bits meet this criterion, returns @size. + */ +static inline +unsigned long find_first_one_and_zero_bit(const unsigned long *addr1, + const unsigned long *addr2, + unsigned long size) +{ + if (small_const_nbits(size)) { + unsigned long val =3D *addr1 & ~*addr2 & GENMASK(size - 1, 0); + + return val ? __ffs(val) : size; + } + + return _find_next_bit(addr1, addr2, size, 0, 0UL, ~0UL, 0); +} +#endif + +#ifndef find_first_zero_and_zero_bit +/** + * find_first_zero_and_zero_bit - find the first bit which is zero in addr= 1 and addr2 memory regions + * @addr1: The first address to base the search on + * @addr2: The second address to base the search on + * @size: The bitmap size in bits + * + * Returns the bit number for the first bit cleared in addr1 and addr2 + * If no corresponding bits meet this criterion, returns @size. + */ +static inline +unsigned long find_first_zero_and_zero_bit(const unsigned long *addr1, + const unsigned long *addr2, + unsigned long size) +{ + if (small_const_nbits(size)) { + unsigned long val =3D ~*addr1 & ~*addr2 & GENMASK(size - 1, 0); + + return val ? __ffs(val) : size; + } + + return _find_next_bit(addr1, addr2, size, 0, ~0UL, ~0UL, 0); +} +#endif + #ifndef find_last_bit /** * find_last_bit - find the last set bit in a memory region --=20 2.17.1 From nobody Tue Apr 14 22:46:25 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8750C00144 for ; Fri, 29 Jul 2022 19:03:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238318AbiG2TDJ (ORCPT ); Fri, 29 Jul 2022 15:03:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51114 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238588AbiG2TCy (ORCPT ); Fri, 29 Jul 2022 15:02:54 -0400 Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9AC9488E24; Fri, 29 Jul 2022 12:02:49 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 9DB8F318B5E; Fri, 29 Jul 2022 15:02:48 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id y4PqTlgBnJHS; Fri, 29 Jul 2022 15:02:48 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id C4461319089; Fri, 29 Jul 2022 15:02:44 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com C4461319089 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1659121364; bh=wX+6cXKANeTzmsi6b1Rv2x8YVliOIgrKgheBkN7FLYs=; h=From:To:Date:Message-Id; b=kRucnBt9zeu06kROqMnvMcC1bVK0f06QFXBmnLTF4HaUNlwLqMqNHFr7p9CLVzN0j OelIf3hawqxn0HNh9pXY6Hn/a5wbG6FHZXD/mCqS8g1w9Zi62PhaR6C4N8RA+sAEV7 p160MWsNY9sYuWUqu9XFSytU95Cq6oyPkktedVbElcqPcrXTBsvvZ8D+DLv2YlCWBZ IO9GeM8FixRvrXTM1Y3wJdkVP5Fon6PKTxB3dIf+hrlHSGf/PYDaZhiXrDT4Oj6TM2 0INjMlcLUWM20IVNBV9CBHsuh7YfNtxrHLIDGHmJbKUrytQ3rLJzqVGGHHE0h9dv4P QDu6wJBLzQg0Q== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 9IxLtaW2HgUY; Fri, 29 Jul 2022 15:02:44 -0400 (EDT) Received: from localhost.localdomain (192-222-180-24.qc.cable.ebox.net [192.222.180.24]) by mail.efficios.com (Postfix) with ESMTPSA id 2906C318A6B; Fri, 29 Jul 2022 15:02:43 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Thomas Gleixner , "Paul E . McKenney" , Boqun Feng , "H . Peter Anvin" , Paul Turner , linux-api@vger.kernel.org, Christian Brauner , Florian Weimer , David.Laight@ACULAB.COM, carlos@redhat.com, Peter Oskolkov , Mathieu Desnoyers Subject: [PATCH v3 08/23] cpumask: implement cpumask_{first,next}_{zero,one}_and_zero Date: Fri, 29 Jul 2022 15:02:10 -0400 Message-Id: <20220729190225.12726-9-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> References: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Allow finding the first or next bit within two input cpumasks which is either: - both zero and zero, - respectively one and zero. Signed-off-by: Mathieu Desnoyers --- include/linux/cpumask.h | 86 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+) diff --git a/include/linux/cpumask.h b/include/linux/cpumask.h index fe29ac7cc469..0f5c3e47423f 100644 --- a/include/linux/cpumask.h +++ b/include/linux/cpumask.h @@ -134,6 +134,18 @@ static inline unsigned int cpumask_first_and(const str= uct cpumask *srcp1, return 0; } =20 +static inline unsigned int cpumask_first_one_and_zero(const struct cpumask= *srcp1, + const struct cpumask *srcp2) +{ + return 0; +} + +static inline unsigned int cpumask_first_zero_and_zero(const struct cpumas= k *srcp1, + const struct cpumask *srcp2) +{ + return 0; +} + static inline unsigned int cpumask_last(const struct cpumask *srcp) { return 0; @@ -157,6 +169,20 @@ static inline unsigned int cpumask_next_and(int n, return n+1; } =20 +static inline unsigned int cpumask_next_one_and_zero(int n, + const struct cpumask *srcp1, + const struct cpumask *srcp2) +{ + return n+1; +} + +static inline unsigned int cpumask_next_zero_and_zero(int n, + const struct cpumask *srcp1, + const struct cpumask *srcp2) +{ + return n+1; +} + static inline unsigned int cpumask_next_wrap(int n, const struct cpumask *= mask, int start, bool wrap) { @@ -230,6 +256,32 @@ unsigned int cpumask_first_and(const struct cpumask *s= rcp1, const struct cpumask return find_first_and_bit(cpumask_bits(srcp1), cpumask_bits(srcp2), nr_cp= umask_bits); } =20 +/** + * cpumask_first_one_and_zero - return the first cpu from *srcp1 & ~*srcp2 + * @src1p: the first input + * @src2p: the second input + * + * Returns >=3D nr_cpu_ids if no cpus match in both. + */ +static inline +unsigned int cpumask_first_one_and_zero(const struct cpumask *srcp1, const= struct cpumask *srcp2) +{ + return find_first_one_and_zero_bit(cpumask_bits(srcp1), cpumask_bits(srcp= 2), nr_cpumask_bits); +} + +/** + * cpumask_first_zero_and_zero - return the first cpu from ~*srcp1 & ~*src= p2 + * @src1p: the first input + * @src2p: the second input + * + * Returns >=3D nr_cpu_ids if no cpus match in both. + */ +static inline +unsigned int cpumask_first_zero_and_zero(const struct cpumask *srcp1, cons= t struct cpumask *srcp2) +{ + return find_first_zero_and_zero_bit(cpumask_bits(srcp1), cpumask_bits(src= p2), nr_cpumask_bits); +} + /** * cpumask_last - get the last CPU in a cpumask * @srcp: - the cpumask pointer @@ -258,6 +310,40 @@ static inline unsigned int cpumask_next_zero(int n, co= nst struct cpumask *srcp) return find_next_zero_bit(cpumask_bits(srcp), nr_cpumask_bits, n+1); } =20 +/** + * cpumask_next_one_and_zero - return the next cpu from *srcp1 & ~*srcp2 + * @n: the cpu prior to the place to search (ie. return will be > @n) + * @src1p: the first input + * @src2p: the second input + * + * Returns >=3D nr_cpu_ids if no cpus match in both. + */ +static inline +unsigned int cpumask_next_one_and_zero(int n, const struct cpumask *srcp1,= const struct cpumask *srcp2) +{ + /* -1 is a legal arg here. */ + if (n !=3D -1) + cpumask_check(n); + return find_next_one_and_zero_bit(cpumask_bits(srcp1), cpumask_bits(srcp2= ), nr_cpumask_bits, n+1); +} + +/** + * cpumask_next_zero_and_zero - return the next cpu from ~*srcp1 & ~*srcp2 + * @n: the cpu prior to the place to search (ie. return will be > @n) + * @src1p: the first input + * @src2p: the second input + * + * Returns >=3D nr_cpu_ids if no cpus match in both. + */ +static inline +unsigned int cpumask_next_zero_and_zero(int n, const struct cpumask *srcp1= , const struct cpumask *srcp2) +{ + /* -1 is a legal arg here. */ + if (n !=3D -1) + cpumask_check(n); + return find_next_zero_and_zero_bit(cpumask_bits(srcp1), cpumask_bits(srcp= 2), nr_cpumask_bits, n+1); +} + int __pure cpumask_next_and(int n, const struct cpumask *, const struct cp= umask *); int __pure cpumask_any_but(const struct cpumask *mask, unsigned int cpu); unsigned int cpumask_local_spread(unsigned int i, int node); --=20 2.17.1 From nobody Tue Apr 14 22:46:25 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 590BFC00144 for ; Fri, 29 Jul 2022 19:03:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238757AbiG2TDd (ORCPT ); Fri, 29 Jul 2022 15:03:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51258 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238617AbiG2TC6 (ORCPT ); Fri, 29 Jul 2022 15:02:58 -0400 Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31CCE88E38; Fri, 29 Jul 2022 12:02:51 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 28A8D318DE3; Fri, 29 Jul 2022 15:02:50 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id mgdsA_sZ0BZA; Fri, 29 Jul 2022 15:02:48 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id C982231908B; Fri, 29 Jul 2022 15:02:44 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com C982231908B DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1659121364; bh=wyoOUTxdpqKzkS37TGZ2RRMdnvchYnZ8+hmrMcS8M24=; h=From:To:Date:Message-Id; b=leA6q5660qhjRTI+M57xA5EcNIXERO79nQA/MBv3BUSDxZE8klkldAF+yFfYDaYP4 +Fh+0G+N6RLdObK7WNOrFc0nl1uRR2arH9JLn1G0CJVMLdUjldb5pYMHt7F3XCGcFX /N+b2lZ4RFmEFjFGVk/GhrBSPVdU7p98DYtJA04VD85BHTqmePSAclWzDls48zCMKE jDeHugIp9Z6U9ZKS1FNQXnZHmQ8IRQR24iOSCIhMPOs0KghA9zSOW44yHJTwhGhp11 DP2eW1EaGgvaBJV4MZH/lZWFYQ5CWi5Kr9KubOVrVcf85XptkCI1Oq2UQmkv5PukWG MfEbkSEe7xTuw== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id p4xmls-ygVBa; Fri, 29 Jul 2022 15:02:44 -0400 (EDT) Received: from localhost.localdomain (192-222-180-24.qc.cable.ebox.net [192.222.180.24]) by mail.efficios.com (Postfix) with ESMTPSA id 87C1F318D78; Fri, 29 Jul 2022 15:02:43 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Thomas Gleixner , "Paul E . McKenney" , Boqun Feng , "H . Peter Anvin" , Paul Turner , linux-api@vger.kernel.org, Christian Brauner , Florian Weimer , David.Laight@ACULAB.COM, carlos@redhat.com, Peter Oskolkov , Mathieu Desnoyers Subject: [PATCH v3 09/23] sched: Introduce per memory space current virtual cpu id Date: Fri, 29 Jul 2022 15:02:11 -0400 Message-Id: <20220729190225.12726-10-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> References: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This feature allows the scheduler to expose a current virtual cpu id to user-space. This virtual cpu id is within the possible cpus range, and is temporarily (and uniquely) assigned while threads are actively running within a memory space. If a memory space has fewer threads than cores, or is limited to run on few cores concurrently through sched affinity or cgroup cpusets, the virtual cpu ids will be values close to 0, thus allowing efficient use of user-space memory for per-cpu data structures. The vcpu_ids are NUMA-aware. On NUMA systems, when a vcpu_id is observed by user-space to be associated with a NUMA node, it is guaranteed to never change NUMA node unless a kernel-level NUMA configuration change happens. This feature is meant to be exposed by a new rseq thread area field. The primary purpose of this feature is to do the heavy-lifting needed by memory allocators to allow them to use per-cpu data structures efficiently in the following situations: - Single-threaded applications, - Multi-threaded applications on large systems (many cores) with limited cpu affinity mask, - Multi-threaded applications on large systems (many cores) with restricted cgroup cpuset per container, - Processes using memory from many NUMA nodes. One of the key concern from scheduler maintainers is the overhead associated with additional atomic operations in the scheduler fast-path. This is why the following optimizations are implemented: 1) On context switch between threads belonging to the same memory space, transfer the mm_vcpu_id from prev to next without any atomic ops. This takes care of use-cases involving frequent context switch between threads belonging to the same memory space. 2) Threads belonging to a memory space with single user (mm_users=3D=3D1) can be assigned mm_vcpu_id=3D0 without any atomic operation on the scheduler fast-path. In non-NUMA, when a memory space goes from single to multi-threaded, lazily allocate the vcpu_id 0 in the mm vcpu mask. This takes care of all single-threaded use-cases involving context switching between threads belonging to different memory spaces. With NUMA, the single-threaded memory space scenario is still special-cased to eliminate all atomic operations on the fast path, but rather than returning vcpu_id=3D0, return the current numa_node_id to allow single-threaded memory spaces to keep good numa locality. On systems where the number of cpus ids is lower than the number of numa node ids, pick the first cpu in the node cpumask rather than the node ID. 3) Introduce a per-runqueue cache containing { mm, vcpu_id } entries. Keep track of the recently allocated vcpu_id for each mm rather than freeing them immediately. This eliminates most atomic ops when context switching back and forth between threads belonging to different memory spaces in multi-threaded scenarios (many processes, each with many threads). The credit goes to Paul Turner (Google) for the vcpu_id idea. This feature is implemented based on the discussions with Paul Turner and Peter Oskolkov (Google), but I took the liberty to implement scheduler fast-path optimizations and my own NUMA-awareness scheme. The rumor has it that Google have been running a rseq vcpu_id extension internally at Google in production for a year. The tcmalloc source code indeed has comments hinting at a vcpu_id prototype extension to the rseq system call [1]. schedstats: * perf bench sched messaging (single instance, multi-process): On sched-switch: single-threaded vcpu-id: 99.985 % transfer between threads: 0 % runqueue cache hit: 0.015 % runqueue cache eviction (bit-clear): 0 % runqueue cache discard (bit-clear): 0 % vcpu-id allocation (bit-set): 0 % On release mm: vcpu-id remove (bit-clear): 0 % On migration: vcpu-id remove (bit-clear): 0 % * perf bench sched messaging -t (single instance, multi-thread): On sched-switch: single-threaded vcpu-id: 0.128 % transfer between threads: 98.260 % runqueue cache hit: 1.075 % runqueue cache eviction (bit-clear): 0.001 % runqueue cache discard (bit-clear): 0 % vcpu-id allocation (bit-set): 0.269 % On release mm: vcpu-id remove (bit-clear): 0.161 % On migration: vcpu-id remove (bit-clear): 0.107 % * perf bench sched messaging -t (two instances, multi-thread): On sched-switch: single-threaded vcpu-id: 0.081 % transfer between threads: 89.512 % runqueue cache hit: 9.659 % runqueue cache eviction (bit-clear): 0.003 % runqueue cache discard (bit-clear): 0 % vcpu-id allocation (bit-set): 0.374 % On release mm: vcpu-id remove (bit-clear): 0.243 % On migration: vcpu-id remove (bit-clear): 0.129 % * perf bench sched pipe (one instance, multi-process): On sched-switch: single-threaded vcpu-id: 99.993 % transfer between threads: 0.001 % runqueue cache hit: 0.002 % runqueue cache eviction (bit-clear): 0 % runqueue cache discard (bit-clear): 0 % vcpu-id allocation (bit-set): 0.002 % On release mm: vcpu-id remove (bit-clear): 0 % On migration: vcpu-id remove (bit-clear): 0.002 % [1] https://github.com/google/tcmalloc/blob/master/tcmalloc/internal/linux_= syscall_support.h#L26 Signed-off-by: Mathieu Desnoyers --- fs/exec.c | 4 + include/linux/mm.h | 25 +++ include/linux/mm_types.h | 111 ++++++++++++ include/linux/sched.h | 5 + init/Kconfig | 4 + kernel/fork.c | 15 +- kernel/sched/core.c | 82 +++++++++ kernel/sched/deadline.c | 3 + kernel/sched/debug.c | 13 ++ kernel/sched/fair.c | 1 + kernel/sched/rt.c | 2 + kernel/sched/sched.h | 357 +++++++++++++++++++++++++++++++++++++++ kernel/sched/stats.c | 16 +- 13 files changed, 635 insertions(+), 3 deletions(-) diff --git a/fs/exec.c b/fs/exec.c index 5a75e92b1a0a..dcd4edd565c4 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -1011,6 +1011,10 @@ static int exec_mmap(struct mm_struct *mm) active_mm =3D tsk->active_mm; tsk->active_mm =3D mm; tsk->mm =3D mm; + mm_init_vcpu_users(mm); + mm_init_vcpumask(mm); + mm_init_node_vcpumask(mm); + sched_vcpu_activate_mm(tsk, mm); /* * This prevents preemption while active_mm is being loaded and * it and mm are being updated, which could cause problems for diff --git a/include/linux/mm.h b/include/linux/mm.h index da08cce2a9fa..cea172092ddf 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3392,4 +3392,29 @@ madvise_set_anon_name(struct mm_struct *mm, unsigned= long start, } #endif =20 +#ifdef CONFIG_SCHED_MM_VCPU +void sched_vcpu_release_mm(struct task_struct *t, struct mm_struct *mm); +void sched_vcpu_activate_mm(struct task_struct *t, struct mm_struct *mm); +void sched_vcpu_get_mm(struct task_struct *t, struct mm_struct *mm); +void sched_vcpu_dup_mm(struct task_struct *t, struct mm_struct *mm); +static inline int task_mm_vcpu_id(struct task_struct *t) +{ + return t->mm_vcpu; +} +#else +static inline void sched_vcpu_release_mm(struct task_struct *t, struct mm_= struct *mm) { } +static inline void sched_vcpu_activate_mm(struct task_struct *t, struct mm= _struct *mm) { } +static inline void sched_vcpu_get_mm(struct task_struct *t, struct mm_stru= ct *mm) { } +static inline void sched_vcpu_dup_mm(struct task_struct *t, struct mm_stru= ct *mm) { } +static inline int task_mm_vcpu_id(struct task_struct *t) +{ + /* + * Use the processor id as a fall-back when the mm vcpu feature is + * disabled. This provides functional per-cpu data structure accesses + * in user-space, althrough it won't provide the memory usage benefits. + */ + return raw_smp_processor_id(); +} +#endif + #endif /* _LINUX_MM_H */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 8834e38c06a4..7854ab5b51f4 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -17,6 +17,7 @@ #include #include #include +#include =20 #include =20 @@ -524,6 +525,20 @@ struct mm_struct { */ atomic_t mm_count; =20 +#ifdef CONFIG_SCHED_MM_VCPU + /** + * @mm_vcpu_users: The number of references to &struct mm_struct + * from user-space threads. + * + * Initialized to 1 for the first thread with a reference to + * the mm. Incremented for each thread getting a reference to the + * mm, and decremented on mm release from user-space threads. + * Used to enable single-threaded mm_vcpu accounting (when =3D=3D 1). + */ + + atomic_t mm_vcpu_users; +#endif + #ifdef CONFIG_MMU atomic_long_t pgtables_bytes; /* PTE page table pages */ #endif @@ -681,6 +696,102 @@ static inline cpumask_t *mm_cpumask(struct mm_struct = *mm) return (struct cpumask *)&mm->cpu_bitmap; } =20 +#ifdef CONFIG_SCHED_MM_VCPU +/* Future-safe accessor for struct mm_struct's vcpu_mask. */ +static inline cpumask_t *mm_vcpumask(struct mm_struct *mm) +{ + unsigned long vcpu_bitmap =3D (unsigned long)mm; + + vcpu_bitmap +=3D offsetof(struct mm_struct, cpu_bitmap); + /* Skip cpu_bitmap */ + vcpu_bitmap +=3D cpumask_size(); + return (struct cpumask *)vcpu_bitmap; +} + +static inline void mm_init_vcpu_users(struct mm_struct *mm) +{ + atomic_set(&mm->mm_vcpu_users, 1); +} + +static inline void mm_init_vcpumask(struct mm_struct *mm) +{ + cpumask_clear(mm_vcpumask(mm)); +} + +static inline unsigned int mm_vcpumask_size(void) +{ + return cpumask_size(); +} + +#else +static inline cpumask_t *mm_vcpumask(struct mm_struct *mm) +{ + return NULL; +} + +static inline void mm_init_vcpu_users(struct mm_struct *mm) { } +static inline void mm_init_vcpumask(struct mm_struct *mm) { } + +static inline unsigned int mm_vcpumask_size(void) +{ + return 0; +} +#endif + +#if defined(CONFIG_SCHED_MM_VCPU) && defined(CONFIG_NUMA) +/* + * Layout of node vcpumasks: + * - node_alloc vcpumask: cpumask tracking which vcpu_id were + * allocated (across nodes) in this + * memory space. + * - node vcpumask[nr_node_ids]: per-node cpumask tracking which vcpu_id + * were allocated in this memory space. + */ +static inline cpumask_t *mm_node_alloc_vcpumask(struct mm_struct *mm) +{ + unsigned long vcpu_bitmap =3D (unsigned long)mm_vcpumask(mm); + + /* Skip mm_vcpumask */ + vcpu_bitmap +=3D cpumask_size(); + return (struct cpumask *)vcpu_bitmap; +} + +static inline cpumask_t *mm_node_vcpumask(struct mm_struct *mm, unsigned i= nt node) +{ + unsigned long vcpu_bitmap =3D (unsigned long)mm_node_alloc_vcpumask(mm); + + /* Skip node alloc vcpumask */ + vcpu_bitmap +=3D cpumask_size(); + vcpu_bitmap +=3D node * cpumask_size(); + return (struct cpumask *)vcpu_bitmap; +} + +static inline void mm_init_node_vcpumask(struct mm_struct *mm) +{ + unsigned int node; + + if (num_possible_nodes() =3D=3D 1) + return; + cpumask_clear(mm_node_alloc_vcpumask(mm)); + for (node =3D 0; node < nr_node_ids; node++) + cpumask_clear(mm_node_vcpumask(mm, node)); +} + +static inline unsigned int mm_node_vcpumask_size(void) +{ + if (num_possible_nodes() =3D=3D 1) + return 0; + return (nr_node_ids + 1) * cpumask_size(); +} +#else +static inline void mm_init_node_vcpumask(struct mm_struct *mm) { } + +static inline unsigned int mm_node_vcpumask_size(void) +{ + return 0; +} +#endif + struct mmu_gather; extern void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm); extern void tlb_gather_mmu_fullmm(struct mmu_gather *tlb, struct mm_struct= *mm); diff --git a/include/linux/sched.h b/include/linux/sched.h index 68b23937b4a5..14f102c2fcfd 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1299,6 +1299,11 @@ struct task_struct { unsigned long rseq_event_mask; #endif =20 +#ifdef CONFIG_SCHED_MM_VCPU + int mm_vcpu; /* Current vcpu in mm */ + int vcpu_mm_active; +#endif + struct tlbflush_unmap_batch tlb_ubc; =20 union { diff --git a/init/Kconfig b/init/Kconfig index fa63cc019ebf..457211f1ca06 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -1041,6 +1041,10 @@ config RT_GROUP_SCHED =20 endif #CGROUP_SCHED =20 +config SCHED_MM_VCPU + def_bool y + depends on SMP && RSEQ + config UCLAMP_TASK_GROUP bool "Utilization clamping per group of tasks" depends on CGROUP_SCHED diff --git a/kernel/fork.c b/kernel/fork.c index 0d8abfb9e0f4..ef5a72d87cee 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1046,6 +1046,11 @@ static struct task_struct *dup_task_struct(struct ta= sk_struct *orig, int node) #ifdef CONFIG_MEMCG tsk->active_memcg =3D NULL; #endif + +#ifdef CONFIG_SCHED_MM_VCPU + tsk->mm_vcpu =3D 0; + tsk->vcpu_mm_active =3D 0; +#endif return tsk; =20 free_stack: @@ -1149,6 +1154,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm= , struct task_struct *p, goto fail_nocontext; =20 mm->user_ns =3D get_user_ns(user_ns); + mm_init_vcpu_users(mm); + mm_init_vcpumask(mm); + mm_init_node_vcpumask(mm); return mm; =20 fail_nocontext: @@ -1450,6 +1458,8 @@ static int wait_for_vfork_done(struct task_struct *ch= ild, */ static void mm_release(struct task_struct *tsk, struct mm_struct *mm) { + sched_vcpu_release_mm(tsk, mm); + uprobe_free_utask(tsk); =20 /* Get rid of any cached register state */ @@ -1569,10 +1579,12 @@ static int copy_mm(unsigned long clone_flags, struc= t task_struct *tsk) if (clone_flags & CLONE_VM) { mmget(oldmm); mm =3D oldmm; + sched_vcpu_get_mm(tsk, mm); } else { mm =3D dup_mm(tsk, current->mm); if (!mm) return -ENOMEM; + sched_vcpu_dup_mm(tsk, mm); } =20 tsk->mm =3D mm; @@ -3003,7 +3015,8 @@ void __init proc_caches_init(void) * dynamically sized based on the maximum CPU number this system * can have, taking hotplug into account (nr_cpu_ids). */ - mm_size =3D sizeof(struct mm_struct) + cpumask_size(); + mm_size =3D sizeof(struct mm_struct) + cpumask_size() + mm_vcpumask_size(= ) + + mm_node_vcpumask_size(); =20 mm_cachep =3D kmem_cache_create_usercopy("mm_struct", mm_size, ARCH_MIN_MMSTRUCT_ALIGN, diff --git a/kernel/sched/core.c b/kernel/sched/core.c index dd11daa7a84b..f7bd328ebb24 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2331,6 +2331,7 @@ static struct rq *move_queued_task(struct rq *rq, str= uct rq_flags *rf, lockdep_assert_rq_held(rq); =20 deactivate_task(rq, p, DEQUEUE_NOCLOCK); + rq_vcpu_cache_remove_mm_locked(rq, p->mm, false); set_task_cpu(p, new_cpu); rq_unlock(rq, rf); =20 @@ -2518,6 +2519,7 @@ int push_cpu_stop(void *arg) // XXX validate p is still the highest prio task if (task_rq(p) =3D=3D rq) { deactivate_task(rq, p, 0); + rq_vcpu_cache_remove_mm_locked(rq, p->mm, false); set_task_cpu(p, lowest_rq->cpu); activate_task(lowest_rq, p, 0); resched_curr(lowest_rq); @@ -3157,6 +3159,7 @@ static void __migrate_swap_task(struct task_struct *p= , int cpu) rq_pin_lock(dst_rq, &drf); =20 deactivate_task(src_rq, p, 0); + rq_vcpu_cache_remove_mm_locked(src_rq, p->mm, false); set_task_cpu(p, cpu); activate_task(dst_rq, p, 0); check_preempt_curr(dst_rq, p, 0); @@ -3780,6 +3783,8 @@ static void __ttwu_queue_wakelist(struct task_struct = *p, int cpu, int wake_flags p->sched_remote_wakeup =3D !!(wake_flags & WF_MIGRATED); =20 WRITE_ONCE(rq->ttwu_pending, 1); + if (WARN_ON_ONCE(task_cpu(p) !=3D cpu_of(rq))) + rq_vcpu_cache_remove_mm(task_rq(p), p->mm, false); __smp_call_single_queue(cpu, &p->wake_entry.llist); } =20 @@ -4189,6 +4194,7 @@ try_to_wake_up(struct task_struct *p, unsigned int st= ate, int wake_flags) =20 wake_flags |=3D WF_MIGRATED; psi_ttwu_dequeue(p); + rq_vcpu_cache_remove_mm(task_rq(p), p->mm, false); set_task_cpu(p, cpu); } #else @@ -4912,6 +4918,7 @@ prepare_task_switch(struct rq *rq, struct task_struct= *prev, sched_info_switch(rq, prev, next); perf_event_task_sched_out(prev, next); rseq_preempt(prev); + switch_mm_vcpu(rq, prev, next); fire_sched_out_preempt_notifiers(prev, next); kmap_local_sched_out(); prepare_task(next); @@ -6043,6 +6050,7 @@ static bool try_steal_cookie(int this, int that) goto next; =20 deactivate_task(src, p, 0); + rq_vcpu_cache_remove_mm_locked(src, p->mm, false); set_task_cpu(p, this); activate_task(dst, p, 0); =20 @@ -11109,3 +11117,77 @@ void call_trace_sched_update_nr_running(struct rq = *rq, int count) { trace_sched_update_nr_running_tp(rq, count); } + +#ifdef CONFIG_SCHED_MM_VCPU +void sched_vcpu_release_mm(struct task_struct *t, struct mm_struct *mm) +{ + struct rq_flags rf; + struct rq *rq; + + if (!mm) + return; + WARN_ON_ONCE(t !=3D current); + preempt_disable(); + rq =3D this_rq(); + rq_lock_irqsave(rq, &rf); + t->vcpu_mm_active =3D 0; + atomic_dec(&mm->mm_vcpu_users); + rq_vcpu_cache_remove_mm_locked(rq, mm, true); + rq_unlock_irqrestore(rq, &rf); + t->mm_vcpu =3D -1; + preempt_enable(); +} + +void sched_vcpu_activate_mm(struct task_struct *t, struct mm_struct *mm) +{ + WARN_ON_ONCE(t !=3D current); + preempt_disable(); + t->vcpu_mm_active =3D 1; + /* No need to reserve in cpumask because single-threaded. */ + t->mm_vcpu =3D mm_vcpu_first_node_vcpu(numa_node_id()); + preempt_enable(); +} + +void sched_vcpu_get_mm(struct task_struct *t, struct mm_struct *mm) +{ + int vcpu, mm_vcpu_users; + struct rq_flags rf; + struct rq *rq; + + preempt_disable(); + rq =3D this_rq(); + t->vcpu_mm_active =3D 1; + mm_vcpu_users =3D atomic_read(&mm->mm_vcpu_users); + atomic_inc(&mm->mm_vcpu_users); + t->mm_vcpu =3D -1; + vcpu =3D current->mm_vcpu; + rq_lock_irqsave(rq, &rf); + /* On transition from 1 to 2 mm users, reserve vcpu ids. */ + if (mm_vcpu_users =3D=3D 1) { + mm_vcpu_reserve_nodes(mm); + rq_vcpu_cache_remove_mm_locked(rq, mm, true); + current->mm_vcpu =3D __mm_vcpu_get(rq, mm); + rq_vcpu_cache_add(rq, mm, current->mm_vcpu); + /* + * __mm_vcpu_get could get a different vcpu after going + * multi-threaded, then back single-threaded, then + * multi-threaded on a NUMA configuration using the first CPU + * matching the NUMA node as single-threaded vcpu, with + * leftover vcpu_id matching the NUMA node set from when this + * task was multithreaded. + */ + if (current->mm_vcpu !=3D vcpu) + rseq_set_notify_resume(current); + } + rq_unlock_irqrestore(rq, &rf); + preempt_enable(); +} + +void sched_vcpu_dup_mm(struct task_struct *t, struct mm_struct *mm) +{ + preempt_disable(); + t->vcpu_mm_active =3D 1; + t->mm_vcpu =3D -1; + preempt_enable(); +} +#endif diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index b61281d10458..20bbb56d554b 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -693,6 +693,7 @@ static struct rq *dl_task_offline_migration(struct rq *= rq, struct task_struct *p __dl_add(dl_b, p->dl.dl_bw, cpumask_weight(later_rq->rd->span)); raw_spin_unlock(&dl_b->lock); =20 + rq_vcpu_cache_remove_mm_locked(rq, p->mm, false); set_task_cpu(p, later_rq->cpu); double_unlock_balance(later_rq, rq); =20 @@ -2319,6 +2320,7 @@ static int push_dl_task(struct rq *rq) } =20 deactivate_task(rq, next_task, 0); + rq_vcpu_cache_remove_mm_locked(rq, next_task->mm, false); set_task_cpu(next_task, later_rq->cpu); =20 /* @@ -2415,6 +2417,7 @@ static void pull_dl_task(struct rq *this_rq) push_task =3D get_push_task(src_rq); } else { deactivate_task(src_rq, p, 0); + rq_vcpu_cache_remove_mm_locked(src_rq, p->mm, false); set_task_cpu(p, this_cpu); activate_task(this_rq, p, 0); dmin =3D p->dl.deadline; diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c index bb3d63bdf4ae..d2f6a6b25a88 100644 --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -762,6 +762,19 @@ do { \ P(sched_goidle); P(ttwu_count); P(ttwu_local); +#undef P +#define P(n) SEQ_printf(m, " .%-30s: %Ld\n", #n, schedstat_val(rq->n)); + P(nr_vcpu_thread_transfer); + P(nr_vcpu_cache_hit); + P(nr_vcpu_cache_evict); + P(nr_vcpu_cache_discard_wrong_node); + P(nr_vcpu_allocate); + P(nr_vcpu_allocate_node_reuse); + P(nr_vcpu_allocate_node_new); + P(nr_vcpu_allocate_node_rebalance); + P(nr_vcpu_allocate_node_steal); + P(nr_vcpu_remove_release_mm); + P(nr_vcpu_remove_migrate); } #undef P =20 diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index cc8daa3dcc8b..941cc84c5ed8 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7883,6 +7883,7 @@ static void detach_task(struct task_struct *p, struct= lb_env *env) lockdep_assert_rq_held(env->src_rq); =20 deactivate_task(env->src_rq, p, DEQUEUE_NOCLOCK); + rq_vcpu_cache_remove_mm_locked(env->src_rq, p->mm, false); set_task_cpu(p, env->dst_cpu); } =20 diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index 7891c0f0e1ff..90321de2bda0 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -2104,6 +2104,7 @@ static int push_rt_task(struct rq *rq, bool pull) } =20 deactivate_task(rq, next_task, 0); + rq_vcpu_cache_remove_mm_locked(rq, next_task->mm, false); set_task_cpu(next_task, lowest_rq->cpu); activate_task(lowest_rq, next_task, 0); resched_curr(lowest_rq); @@ -2377,6 +2378,7 @@ static void pull_rt_task(struct rq *this_rq) push_task =3D get_push_task(src_rq); } else { deactivate_task(src_rq, p, 0); + rq_vcpu_cache_remove_mm_locked(src_rq, p->mm, false); set_task_cpu(p, this_cpu); activate_task(this_rq, p, 0); resched =3D true; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 84bba67c92dc..cc621f4691ba 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -901,6 +901,19 @@ struct uclamp_rq { DECLARE_STATIC_KEY_FALSE(sched_uclamp_used); #endif /* CONFIG_UCLAMP_TASK */ =20 +#ifdef CONFIG_SCHED_MM_VCPU +# define RQ_VCPU_CACHE_SIZE 8 +struct rq_vcpu_entry { + struct mm_struct *mm; /* NULL if unset */ + int vcpu_id; +}; + +struct rq_vcpu_cache { + struct rq_vcpu_entry entry[RQ_VCPU_CACHE_SIZE]; + unsigned int head; +}; +#endif + /* * This is the main, per-CPU runqueue data structure. * @@ -1071,6 +1084,19 @@ struct rq { /* try_to_wake_up() stats */ unsigned int ttwu_count; unsigned int ttwu_local; + + unsigned long long nr_vcpu_single_thread; + unsigned long long nr_vcpu_thread_transfer; + unsigned long long nr_vcpu_cache_hit; + unsigned long long nr_vcpu_cache_evict; + unsigned long long nr_vcpu_cache_discard_wrong_node; + unsigned long long nr_vcpu_allocate; + unsigned long long nr_vcpu_allocate_node_reuse; + unsigned long long nr_vcpu_allocate_node_new; + unsigned long long nr_vcpu_allocate_node_rebalance; + unsigned long long nr_vcpu_allocate_node_steal; + unsigned long long nr_vcpu_remove_release_mm; + unsigned long long nr_vcpu_remove_migrate; #endif =20 #ifdef CONFIG_CPU_IDLE @@ -1101,6 +1127,10 @@ struct rq { unsigned int core_forceidle_occupation; u64 core_forceidle_start; #endif + +#ifdef CONFIG_SCHED_MM_VCPU + struct rq_vcpu_cache vcpu_cache; +#endif }; =20 #ifdef CONFIG_FAIR_GROUP_SCHED @@ -3112,4 +3142,331 @@ extern int sched_dynamic_mode(const char *str); extern void sched_dynamic_update(int mode); #endif =20 +#ifdef CONFIG_SCHED_MM_VCPU +static inline int __mm_vcpu_get_single_node(struct mm_struct *mm) +{ + struct cpumask *cpumask; + int vcpu; + + cpumask =3D mm_vcpumask(mm); + /* Atomically reserve lowest available vcpu number. */ + do { + vcpu =3D cpumask_first_zero(cpumask); + if (vcpu >=3D nr_cpu_ids) + return -1; + } while (cpumask_test_and_set_cpu(vcpu, cpumask)); + return vcpu; +} + +#ifdef CONFIG_NUMA +static inline bool mm_node_vcpumask_test_cpu(struct mm_struct *mm, int vcp= u_id) +{ + if (num_possible_nodes() =3D=3D 1) + return true; + return cpumask_test_cpu(vcpu_id, mm_node_vcpumask(mm, numa_node_id())); +} + +static inline int __mm_vcpu_get(struct rq *rq, struct mm_struct *mm) +{ + struct cpumask *cpumask =3D mm_vcpumask(mm), + *node_cpumask =3D mm_node_vcpumask(mm, numa_node_id()), + *node_alloc_cpumask =3D mm_node_alloc_vcpumask(mm); + unsigned int node; + int vcpu; + + if (num_possible_nodes() =3D=3D 1) + return __mm_vcpu_get_single_node(mm); + + /* + * Try to atomically reserve lowest available vcpu number within those + * already reserved for this NUMA node. + */ + do { + vcpu =3D cpumask_first_one_and_zero(node_cpumask, cpumask); + if (vcpu >=3D nr_cpu_ids) + goto alloc_numa; + } while (cpumask_test_and_set_cpu(vcpu, cpumask)); + schedstat_inc(rq->nr_vcpu_allocate_node_reuse); + goto end; + +alloc_numa: + /* + * Try to atomically reserve lowest available vcpu number within those + * not already allocated for numa nodes. + */ + do { + vcpu =3D cpumask_first_zero_and_zero(node_alloc_cpumask, cpumask); + if (vcpu >=3D nr_cpu_ids) + goto numa_update; + } while (cpumask_test_and_set_cpu(vcpu, cpumask)); + cpumask_set_cpu(vcpu, node_cpumask); + cpumask_set_cpu(vcpu, node_alloc_cpumask); + schedstat_inc(rq->nr_vcpu_allocate_node_new); + goto end; + +numa_update: + /* + * NUMA node id configuration changed for at least one CPU in the system. + * We need to steal a currently unused vcpu_id from an overprovisioned + * node for our current node. Userspace must handle the fact that the + * node id associated with this vcpu_id may change due to node ID + * reconfiguration. + * + * Count how many possible cpus are attached to each (other) node id, + * and compare this with the per-mm node vcpumask cpu count. Find one + * which has too many cpus in its mask to steal from. + */ + for (node =3D 0; node < nr_node_ids; node++) { + struct cpumask *iter_cpumask; + + if (node =3D=3D numa_node_id()) + continue; + iter_cpumask =3D mm_node_vcpumask(mm, node); + if (nr_cpus_node(node) < cpumask_weight(iter_cpumask)) { + /* Try to steal from this node. */ + do { + vcpu =3D cpumask_first_one_and_zero(iter_cpumask, cpumask); + if (vcpu >=3D nr_cpu_ids) + goto steal_fail; + } while (cpumask_test_and_set_cpu(vcpu, cpumask)); + cpumask_clear_cpu(vcpu, iter_cpumask); + cpumask_set_cpu(vcpu, node_cpumask); + schedstat_inc(rq->nr_vcpu_allocate_node_rebalance); + goto end; + } + } + +steal_fail: + /* + * Our attempt at gracefully stealing a vcpu_id from another + * overprovisioned NUMA node failed. Fallback to grabbing the first + * available vcpu_id. + */ + do { + vcpu =3D cpumask_first_zero(cpumask); + if (vcpu >=3D nr_cpu_ids) + return -1; + } while (cpumask_test_and_set_cpu(vcpu, cpumask)); + /* Steal vcpu from its numa node mask. */ + for (node =3D 0; node < nr_node_ids; node++) { + struct cpumask *iter_cpumask; + + if (node =3D=3D numa_node_id()) + continue; + iter_cpumask =3D mm_node_vcpumask(mm, node); + if (cpumask_test_cpu(vcpu, iter_cpumask)) { + cpumask_clear_cpu(vcpu, iter_cpumask); + break; + } + } + cpumask_set_cpu(vcpu, node_cpumask); + schedstat_inc(rq->nr_vcpu_allocate_node_steal); +end: + return vcpu; +} + +static inline int mm_vcpu_first_node_vcpu(int node) +{ + if (likely(nr_cpu_ids >=3D nr_node_ids)) + return node; + else { + int vcpu; + + vcpu =3D cpumask_first(cpumask_of_node(node)); + if (vcpu >=3D nr_cpu_ids) + return -1; + return vcpu; + } +} + +/* + * Single-threaded processes observe a mapping of vcpu_id->node_id where + * the vcpu_id returned corresponds to mm_vcpu_first_node_vcpu(). When goi= ng + * from single to multi-threaded, reserve this same mapping so it stays + * invariant. + */ +static inline void mm_vcpu_reserve_nodes(struct mm_struct *mm) +{ + struct cpumask *node_alloc_cpumask =3D mm_node_alloc_vcpumask(mm); + int node, other_node; + + for (node =3D 0; node < nr_node_ids; node++) { + struct cpumask *iter_cpumask =3D mm_node_vcpumask(mm, node); + int vcpu =3D mm_vcpu_first_node_vcpu(node); + + /* Skip nodes that have no CPU associated with them. */ + if (vcpu < 0) + continue; + cpumask_set_cpu(vcpu, iter_cpumask); + cpumask_set_cpu(vcpu, node_alloc_cpumask); + for (other_node =3D 0; other_node < nr_node_ids; other_node++) { + if (other_node =3D=3D node) + continue; + cpumask_clear_cpu(vcpu, mm_node_vcpumask(mm, other_node)); + } + } +} +#else +static inline bool mm_node_vcpumask_test_cpu(struct mm_struct *mm, int vcp= u_id) +{ + return true; +} +static inline int __mm_vcpu_get(struct rq *rq, struct mm_struct *mm) +{ + return __mm_vcpu_get_single_node(mm); +} +static inline int mm_vcpu_first_node_vcpu(int node) +{ + return 0; +} +static inline void mm_vcpu_reserve_nodes(struct mm_struct *mm) { } +#endif + +static inline void __mm_vcpu_put(struct mm_struct *mm, int vcpu) +{ + if (vcpu < 0) + return; + cpumask_clear_cpu(vcpu, mm_vcpumask(mm)); +} + +static inline struct rq_vcpu_entry *rq_vcpu_cache_lookup(struct rq *rq, st= ruct mm_struct *mm) +{ + struct rq_vcpu_cache *vcpu_cache =3D &rq->vcpu_cache; + int i; + + for (i =3D 0; i < RQ_VCPU_CACHE_SIZE; i++) { + struct rq_vcpu_entry *entry =3D &vcpu_cache->entry[i]; + + if (entry->mm =3D=3D mm) + return entry; + } + return NULL; +} + +/* Removal from cache simply leaves an unused hole. */ +static inline int rq_vcpu_cache_lookup_remove(struct rq *rq, struct mm_str= uct *mm) +{ + struct rq_vcpu_entry *entry =3D rq_vcpu_cache_lookup(rq, mm); + + if (!entry) + return -1; + entry->mm =3D NULL; /* Remove from cache */ + return entry->vcpu_id; +} + +static inline void rq_vcpu_cache_remove_mm_locked(struct rq *rq, struct mm= _struct *mm, bool release_mm) +{ + int vcpu; + + if (!mm) + return; + /* + * Do not remove the cache entry for a runqueue that runs a task which + * currently uses the target mm. + */ + if (!release_mm && rq->curr->mm =3D=3D mm) + return; + vcpu =3D rq_vcpu_cache_lookup_remove(rq, mm); + if (vcpu < 0) + return; + if (release_mm) + schedstat_inc(rq->nr_vcpu_remove_release_mm); + else + schedstat_inc(rq->nr_vcpu_remove_migrate); + __mm_vcpu_put(mm, vcpu); +} + +static inline void rq_vcpu_cache_remove_mm(struct rq *rq, struct mm_struct= *mm, bool release_mm) +{ + struct rq_flags rf; + + rq_lock_irqsave(rq, &rf); + rq_vcpu_cache_remove_mm_locked(rq, mm, release_mm); + rq_unlock_irqrestore(rq, &rf); +} + +/* + * Add at head, move head forward. Cheap LRU cache. + * Only need to clear the vcpu mask bit from its own mm_vcpumask(mm) when = we + * overwrite an old entry from the cache. Note that this is not needed if = the + * overwritten entry is an unused hole. This access to the old_mm from an + * unrelated thread requires that cache entry for a given mm gets pruned f= rom + * the cache when a task is dequeued from the runqueue. + */ +static inline void rq_vcpu_cache_add(struct rq *rq, struct mm_struct *mm, = int vcpu_id) +{ + struct rq_vcpu_cache *vcpu_cache =3D &rq->vcpu_cache; + struct mm_struct *old_mm; + struct rq_vcpu_entry *entry; + unsigned int pos; + + pos =3D vcpu_cache->head; + entry =3D &vcpu_cache->entry[pos]; + old_mm =3D entry->mm; + if (old_mm) { + schedstat_inc(rq->nr_vcpu_cache_evict); + __mm_vcpu_put(old_mm, entry->vcpu_id); + } + entry->mm =3D mm; + entry->vcpu_id =3D vcpu_id; + vcpu_cache->head =3D (pos + 1) % RQ_VCPU_CACHE_SIZE; +} + +static inline int mm_vcpu_get(struct rq *rq, struct task_struct *t) +{ + struct rq_vcpu_entry *entry; + struct mm_struct *mm =3D t->mm; + int vcpu; + + /* Skip allocation if mm is single-threaded. */ + if (atomic_read(&mm->mm_vcpu_users) =3D=3D 1) { + schedstat_inc(rq->nr_vcpu_single_thread); + vcpu =3D mm_vcpu_first_node_vcpu(numa_node_id()); + goto end; + } + entry =3D rq_vcpu_cache_lookup(rq, mm); + if (likely(entry)) { + vcpu =3D entry->vcpu_id; + if (likely(mm_node_vcpumask_test_cpu(mm, vcpu))) { + schedstat_inc(rq->nr_vcpu_cache_hit); + goto end; + } else { + schedstat_inc(rq->nr_vcpu_cache_discard_wrong_node); + entry->mm =3D NULL; /* Remove from cache */ + __mm_vcpu_put(mm, vcpu); + } + } + schedstat_inc(rq->nr_vcpu_allocate); + vcpu =3D __mm_vcpu_get(rq, mm); + rq_vcpu_cache_add(rq, mm, vcpu); +end: + return vcpu; +} + +static inline void switch_mm_vcpu(struct rq *rq, struct task_struct *prev,= struct task_struct *next) +{ + if (!(next->flags & PF_EXITING) && !(next->flags & PF_KTHREAD) && next->m= m && next->vcpu_mm_active) { + if (!(prev->flags & PF_EXITING) && !(prev->flags & PF_KTHREAD) && prev->= mm =3D=3D next->mm && + prev->vcpu_mm_active && + mm_node_vcpumask_test_cpu(next->mm, prev->mm_vcpu)) { + /* + * Switching between threads with the same mm. Simply pass the + * vcpu token along to the next thread. + */ + schedstat_inc(rq->nr_vcpu_thread_transfer); + next->mm_vcpu =3D prev->mm_vcpu; + } else { + next->mm_vcpu =3D mm_vcpu_get(rq, next); + } + } + if (!(prev->flags & PF_EXITING) && !(prev->flags & PF_KTHREAD) && prev->m= m && prev->vcpu_mm_active) + prev->mm_vcpu =3D -1; +} + +#else +static inline void switch_mm_vcpu(struct rq *rq, struct task_struct *prev,= struct task_struct *next) { } +static inline void rq_vcpu_cache_remove_mm_locked(struct rq *rq, struct mm= _struct *mm, bool release_mm) { } +static inline void rq_vcpu_cache_remove_mm(struct rq *rq, struct mm_struct= *mm, bool release_mm) { } +#endif + #endif /* _KERNEL_SCHED_SCHED_H */ diff --git a/kernel/sched/stats.c b/kernel/sched/stats.c index 857f837f52cb..29e168af777b 100644 --- a/kernel/sched/stats.c +++ b/kernel/sched/stats.c @@ -133,12 +133,24 @@ static int show_schedstat(struct seq_file *seq, void = *v) =20 /* runqueue-specific stats */ seq_printf(seq, - "cpu%d %u 0 %u %u %u %u %llu %llu %lu", + "cpu%d %u 0 %u %u %u %u %llu %llu %lu %llu %llu %llu %llu %llu %llu = %llu %llu %llu %llu %llu %llu", cpu, rq->yld_count, rq->sched_count, rq->sched_goidle, rq->ttwu_count, rq->ttwu_local, rq->rq_cpu_time, - rq->rq_sched_info.run_delay, rq->rq_sched_info.pcount); + rq->rq_sched_info.run_delay, rq->rq_sched_info.pcount, + rq->nr_vcpu_single_thread, + rq->nr_vcpu_thread_transfer, + rq->nr_vcpu_cache_hit, + rq->nr_vcpu_cache_evict, + rq->nr_vcpu_cache_discard_wrong_node, + rq->nr_vcpu_allocate, + rq->nr_vcpu_allocate_node_reuse, + rq->nr_vcpu_allocate_node_new, + rq->nr_vcpu_allocate_node_rebalance, + rq->nr_vcpu_allocate_node_steal, + rq->nr_vcpu_remove_release_mm, + rq->nr_vcpu_remove_migrate); =20 seq_printf(seq, "\n"); =20 --=20 2.17.1 From nobody Tue Apr 14 22:46:25 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 415FDC00144 for ; Fri, 29 Jul 2022 19:03:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238738AbiG2TDV (ORCPT ); Fri, 29 Jul 2022 15:03:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51218 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238597AbiG2TC5 (ORCPT ); Fri, 29 Jul 2022 15:02:57 -0400 Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 43FD188E29; Fri, 29 Jul 2022 12:02:50 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 055D431901A; Fri, 29 Jul 2022 15:02:49 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id uCippXD8KXZi; Fri, 29 Jul 2022 15:02:48 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 35903318E98; Fri, 29 Jul 2022 15:02:45 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 35903318E98 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1659121365; bh=SlksVcAEQwHXv/2n1z59vggbF/JR7rR40IRrV+e+NmA=; h=From:To:Date:Message-Id; b=dom/woOiwwNfE+wiiNMkWguctUUrw2JryzhuC7UabwX1Tw1LbQRRc3xehozmHcJvs xwqFcp8epyeqhH107olHOwiMsyC+DH8Dm8jKJMspgHs9KkdFjAF0+8ohR68j7GwwAF TtsveO+8pvg6X4qIN5htU+ZsFQlJv7MOUh08Ac2LBOX4ft9xjI7hyCsi/erlJ0WBr1 aJcwmLZT2QIeiubqVO6RkjXmVTDHnXtYW8UG0no7fBYbwzveVjaXnCRiDRQIFRnJoH gO9GxbMMVPqIXNNenl11ThtU3OC9sReD4cyvUXJEorzT7WluUA61B30cNLjk6TTxUM pUfOpPLBUobBw== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id V-BhCphR6MiV; Fri, 29 Jul 2022 15:02:45 -0400 (EDT) Received: from localhost.localdomain (192-222-180-24.qc.cable.ebox.net [192.222.180.24]) by mail.efficios.com (Postfix) with ESMTPSA id E4E0A318E1A; Fri, 29 Jul 2022 15:02:43 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Thomas Gleixner , "Paul E . McKenney" , Boqun Feng , "H . Peter Anvin" , Paul Turner , linux-api@vger.kernel.org, Christian Brauner , Florian Weimer , David.Laight@ACULAB.COM, carlos@redhat.com, Peter Oskolkov , Mathieu Desnoyers Subject: [PATCH v3 10/23] rseq: extend struct rseq with per memory space vcpu id Date: Fri, 29 Jul 2022 15:02:12 -0400 Message-Id: <20220729190225.12726-11-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> References: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" If a memory space has fewer threads than cores, or is limited to run on few cores concurrently through sched affinity or cgroup cpusets, the virtual cpu ids will be values close to 0, thus allowing efficient use of user-space memory for per-cpu data structures. Signed-off-by: Mathieu Desnoyers --- include/uapi/linux/rseq.h | 9 +++++++++ kernel/rseq.c | 10 +++++++++- 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/include/uapi/linux/rseq.h b/include/uapi/linux/rseq.h index 1cb90a435c5c..77a136586ac6 100644 --- a/include/uapi/linux/rseq.h +++ b/include/uapi/linux/rseq.h @@ -139,6 +139,15 @@ struct rseq { */ __u32 node_id; =20 + /* + * Restartable sequences vm_vcpu_id field. Updated by the kernel. Read by + * user-space with single-copy atomicity semantics. This field should + * only be read by the thread which registered this data structure. + * Aligned on 32-bit. Contains the current thread's virtual CPU ID + * (allocated uniquely within a memory space). + */ + __u32 vm_vcpu_id; + /* * Flexible array member at end of structure, after last feature field. */ diff --git a/kernel/rseq.c b/kernel/rseq.c index cb7d8a5afc82..1b00339c341b 100644 --- a/kernel/rseq.c +++ b/kernel/rseq.c @@ -89,12 +89,14 @@ static int rseq_update_cpu_node_id(struct task_struct *= t) struct rseq __user *rseq =3D t->rseq; u32 cpu_id =3D raw_smp_processor_id(); u32 node_id =3D cpu_to_node(cpu_id); + u32 vm_vcpu_id =3D task_mm_vcpu_id(t); =20 if (!user_write_access_begin(rseq, t->rseq_len)) goto efault; unsafe_put_user(cpu_id, &rseq->cpu_id_start, efault_end); unsafe_put_user(cpu_id, &rseq->cpu_id, efault_end); unsafe_put_user(node_id, &rseq->node_id, efault_end); + unsafe_put_user(vm_vcpu_id, &rseq->vm_vcpu_id, efault_end); /* * Additional feature fields added after ORIG_RSEQ_SIZE * need to be conditionally updated only if @@ -112,7 +114,8 @@ static int rseq_update_cpu_node_id(struct task_struct *= t) =20 static int rseq_reset_rseq_cpu_node_id(struct task_struct *t) { - u32 cpu_id_start =3D 0, cpu_id =3D RSEQ_CPU_ID_UNINITIALIZED, node_id =3D= 0; + u32 cpu_id_start =3D 0, cpu_id =3D RSEQ_CPU_ID_UNINITIALIZED, node_id =3D= 0, + vm_vcpu_id =3D 0; =20 /* * Reset cpu_id_start to its initial state (0). @@ -131,6 +134,11 @@ static int rseq_reset_rseq_cpu_node_id(struct task_str= uct *t) */ if (put_user(node_id, &t->rseq->node_id)) return -EFAULT; + /* + * Reset vm_vcpu_id to its initial state (0). + */ + if (put_user(vm_vcpu_id, &t->rseq->vm_vcpu_id)) + return -EFAULT; /* * Additional feature fields added after ORIG_RSEQ_SIZE * need to be conditionally reset only if --=20 2.17.1 From nobody Tue Apr 14 22:46:25 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2125DC00144 for ; Fri, 29 Jul 2022 19:03:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238748AbiG2TD0 (ORCPT ); Fri, 29 Jul 2022 15:03:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51230 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238526AbiG2TC5 (ORCPT ); Fri, 29 Jul 2022 15:02:57 -0400 Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C38F588E2D; Fri, 29 Jul 2022 12:02:50 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id A9773318FB6; Fri, 29 Jul 2022 15:02:49 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id s8ha9n99vugw; Fri, 29 Jul 2022 15:02:49 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 35F3F318E99; Fri, 29 Jul 2022 15:02:45 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 35F3F318E99 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1659121365; bh=kAz9JX6I21RCSH6EIAOuxWPy1z06YjZt9LbM5+71v4w=; h=From:To:Date:Message-Id; b=WX13L7+TEOaPevZkKTRcECVGvDA07+/s5E5CJIRF4a9vHFA/YRjKzFV0x5HqIcm/H x3d72dyDtZEHOgF21EjCEX41UM19Fu8xrwEhxGlPZiJJswQiWa/yk2WAkmIzfp9LhB q4tCfgciJVcOysCVGPlWoKzMLOFX7h0cYrs/DeWgGq0GQzzpw+cZuI2ReyrhR/Of5D 0HpVUb+9tfj3Wo1j36OP7K2owA53HoXhsCvrsaETtYUS0mNIzG9yXF9u0kDl5/3XnJ H8Qbj5mcKbWUCa7hwgv/Gm1qV3/p9Q7M/vxEtk5IVQdjaLFFtxASKvA0fU8zLwGoIt BraW7JPapAGhQ== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id J4b92CPtDNz8; Fri, 29 Jul 2022 15:02:45 -0400 (EDT) Received: from localhost.localdomain (192-222-180-24.qc.cable.ebox.net [192.222.180.24]) by mail.efficios.com (Postfix) with ESMTPSA id 417CD318D79; Fri, 29 Jul 2022 15:02:44 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Thomas Gleixner , "Paul E . McKenney" , Boqun Feng , "H . Peter Anvin" , Paul Turner , linux-api@vger.kernel.org, Christian Brauner , Florian Weimer , David.Laight@ACULAB.COM, carlos@redhat.com, Peter Oskolkov , Mathieu Desnoyers Subject: [PATCH v3 11/23] selftests/rseq: Remove RSEQ_SKIP_FASTPATH code Date: Fri, 29 Jul 2022 15:02:13 -0400 Message-Id: <20220729190225.12726-12-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> References: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This code is not currently build by the test Makefile, adds complexity, and is not overall useful considering that the abort handling loops to retry the fast-path. Signed-off-by: Mathieu Desnoyers --- tools/testing/selftests/rseq/param_test.c | 4 -- tools/testing/selftests/rseq/rseq-arm.h | 6 --- tools/testing/selftests/rseq/rseq-arm64.h | 6 --- tools/testing/selftests/rseq/rseq-mips.h | 6 --- tools/testing/selftests/rseq/rseq-ppc.h | 6 --- tools/testing/selftests/rseq/rseq-riscv.h | 6 --- tools/testing/selftests/rseq/rseq-s390.h | 5 -- tools/testing/selftests/rseq/rseq-skip.h | 65 ----------------------- tools/testing/selftests/rseq/rseq-x86.h | 12 ----- 9 files changed, 116 deletions(-) delete mode 100644 tools/testing/selftests/rseq/rseq-skip.h diff --git a/tools/testing/selftests/rseq/param_test.c b/tools/testing/self= tests/rseq/param_test.c index ef29bc16f358..9869369a8607 100644 --- a/tools/testing/selftests/rseq/param_test.c +++ b/tools/testing/selftests/rseq/param_test.c @@ -38,11 +38,7 @@ static int opt_yield, opt_signal, opt_sleep, opt_disable_rseq, opt_threads =3D 200, opt_disable_mod =3D 0, opt_test =3D 's', opt_mb =3D 0; =20 -#ifndef RSEQ_SKIP_FASTPATH static long long opt_reps =3D 5000; -#else -static long long opt_reps =3D 100; -#endif =20 static __thread __attribute__((tls_model("initial-exec"))) unsigned int signals_delivered; diff --git a/tools/testing/selftests/rseq/rseq-arm.h b/tools/testing/selfte= sts/rseq/rseq-arm.h index 893a11eca9d5..7445107f842b 100644 --- a/tools/testing/selftests/rseq/rseq-arm.h +++ b/tools/testing/selftests/rseq/rseq-arm.h @@ -79,10 +79,6 @@ do { \ RSEQ_WRITE_ONCE(*p, v); \ } while (0) =20 -#ifdef RSEQ_SKIP_FASTPATH -#include "rseq-skip.h" -#else /* !RSEQ_SKIP_FASTPATH */ - #define __RSEQ_ASM_DEFINE_TABLE(label, version, flags, start_ip, \ post_commit_offset, abort_ip) \ ".pushsection __rseq_cs, \"aw\"\n\t" \ @@ -823,5 +819,3 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, i= ntptr_t expect, rseq_bug("expected value comparison failed"); #endif } - -#endif /* !RSEQ_SKIP_FASTPATH */ diff --git a/tools/testing/selftests/rseq/rseq-arm64.h b/tools/testing/self= tests/rseq/rseq-arm64.h index cbe190a4d005..49c387fcd868 100644 --- a/tools/testing/selftests/rseq/rseq-arm64.h +++ b/tools/testing/selftests/rseq/rseq-arm64.h @@ -85,10 +85,6 @@ do { \ } \ } while (0) =20 -#ifdef RSEQ_SKIP_FASTPATH -#include "rseq-skip.h" -#else /* !RSEQ_SKIP_FASTPATH */ - #define RSEQ_ASM_TMP_REG32 "w15" #define RSEQ_ASM_TMP_REG "x15" #define RSEQ_ASM_TMP_REG_2 "x14" @@ -691,5 +687,3 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, i= ntptr_t expect, rseq_bug("expected value comparison failed"); #endif } - -#endif /* !RSEQ_SKIP_FASTPATH */ diff --git a/tools/testing/selftests/rseq/rseq-mips.h b/tools/testing/selft= ests/rseq/rseq-mips.h index 878739fae2fd..dd199952d649 100644 --- a/tools/testing/selftests/rseq/rseq-mips.h +++ b/tools/testing/selftests/rseq/rseq-mips.h @@ -60,10 +60,6 @@ do { \ RSEQ_WRITE_ONCE(*p, v); \ } while (0) =20 -#ifdef RSEQ_SKIP_FASTPATH -#include "rseq-skip.h" -#else /* !RSEQ_SKIP_FASTPATH */ - #if _MIPS_SZLONG =3D=3D 64 # define LONG ".dword" # define LONG_LA "dla" @@ -773,5 +769,3 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, i= ntptr_t expect, rseq_bug("expected value comparison failed"); #endif } - -#endif /* !RSEQ_SKIP_FASTPATH */ diff --git a/tools/testing/selftests/rseq/rseq-ppc.h b/tools/testing/selfte= sts/rseq/rseq-ppc.h index bab8e0b9fb11..f82d95c1bb3f 100644 --- a/tools/testing/selftests/rseq/rseq-ppc.h +++ b/tools/testing/selftests/rseq/rseq-ppc.h @@ -36,10 +36,6 @@ do { \ RSEQ_WRITE_ONCE(*p, v); \ } while (0) =20 -#ifdef RSEQ_SKIP_FASTPATH -#include "rseq-skip.h" -#else /* !RSEQ_SKIP_FASTPATH */ - /* * The __rseq_cs_ptr_array and __rseq_cs sections can be used by debuggers= to * better handle single-stepping through the restartable critical sections. @@ -787,5 +783,3 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, i= ntptr_t expect, rseq_bug("expected value comparison failed"); #endif } - -#endif /* !RSEQ_SKIP_FASTPATH */ diff --git a/tools/testing/selftests/rseq/rseq-riscv.h b/tools/testing/self= tests/rseq/rseq-riscv.h index b86642f90d7f..3394b8f7f322 100644 --- a/tools/testing/selftests/rseq/rseq-riscv.h +++ b/tools/testing/selftests/rseq/rseq-riscv.h @@ -49,10 +49,6 @@ do { \ RSEQ_WRITE_ONCE(*(p), v); \ } while (0) =20 -#ifdef RSEQ_SKIP_FASTPATH -#include "rseq-skip.h" -#else /* !RSEQ_SKIP_FASTPATH */ - #define __RSEQ_ASM_DEFINE_TABLE(label, version, flags, start_ip, \ post_commit_offset, abort_ip) \ ".pushsection __rseq_cs, \"aw\"\n" \ @@ -673,5 +669,3 @@ int rseq_offset_deref_addv(intptr_t *ptr, off_t off, in= tptr_t inc, int cpu) rseq_bug("cpu_id comparison failed"); #endif } - -#endif /* !RSEQ_SKIP_FASTPATH */ diff --git a/tools/testing/selftests/rseq/rseq-s390.h b/tools/testing/selft= ests/rseq/rseq-s390.h index 4e6dc5f0cb42..4d3286453bbf 100644 --- a/tools/testing/selftests/rseq/rseq-s390.h +++ b/tools/testing/selftests/rseq/rseq-s390.h @@ -28,10 +28,6 @@ do { \ RSEQ_WRITE_ONCE(*p, v); \ } while (0) =20 -#ifdef RSEQ_SKIP_FASTPATH -#include "rseq-skip.h" -#else /* !RSEQ_SKIP_FASTPATH */ - #ifdef __s390x__ =20 #define LONG_L "lg" @@ -607,4 +603,3 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, i= ntptr_t expect, return rseq_cmpeqv_trymemcpy_storev(v, expect, dst, src, len, newv, cpu); } -#endif /* !RSEQ_SKIP_FASTPATH */ diff --git a/tools/testing/selftests/rseq/rseq-skip.h b/tools/testing/selft= ests/rseq/rseq-skip.h deleted file mode 100644 index 7b53dac1fcdd..000000000000 --- a/tools/testing/selftests/rseq/rseq-skip.h +++ /dev/null @@ -1,65 +0,0 @@ -/* SPDX-License-Identifier: LGPL-2.1 OR MIT */ -/* - * rseq-skip.h - * - * (C) Copyright 2017-2018 - Mathieu Desnoyers - */ - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cp= u) -{ - return -1; -} - -static inline __attribute__((always_inline)) -int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, - long voffp, intptr_t *load, int cpu) -{ - return -1; -} - -static inline __attribute__((always_inline)) -int rseq_addv(intptr_t *v, intptr_t count, int cpu) -{ - return -1; -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t newv2, - intptr_t newv, int cpu) -{ - return -1; -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t newv2, - intptr_t newv, int cpu) -{ - return -1; -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t expect2, - intptr_t newv, int cpu) -{ - return -1; -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, - void *dst, void *src, size_t len, - intptr_t newv, int cpu) -{ - return -1; -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, - void *dst, void *src, size_t len, - intptr_t newv, int cpu) -{ - return -1; -} diff --git a/tools/testing/selftests/rseq/rseq-x86.h b/tools/testing/selfte= sts/rseq/rseq-x86.h index bd01dc41ca13..e148dfb2f68a 100644 --- a/tools/testing/selftests/rseq/rseq-x86.h +++ b/tools/testing/selftests/rseq/rseq-x86.h @@ -50,10 +50,6 @@ do { \ RSEQ_WRITE_ONCE(*p, v); \ } while (0) =20 -#ifdef RSEQ_SKIP_FASTPATH -#include "rseq-skip.h" -#else /* !RSEQ_SKIP_FASTPATH */ - #define __RSEQ_ASM_DEFINE_TABLE(label, version, flags, \ start_ip, post_commit_offset, abort_ip) \ ".pushsection __rseq_cs, \"aw\"\n\t" \ @@ -629,8 +625,6 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, i= ntptr_t expect, newv, cpu); } =20 -#endif /* !RSEQ_SKIP_FASTPATH */ - #elif defined(__i386__) =20 #define RSEQ_ASM_TP_SEGMENT %%gs @@ -657,10 +651,6 @@ do { \ RSEQ_WRITE_ONCE(*p, v); \ } while (0) =20 -#ifdef RSEQ_SKIP_FASTPATH -#include "rseq-skip.h" -#else /* !RSEQ_SKIP_FASTPATH */ - /* * Use eax as scratch register and take memory operands as input to * lessen register pressure. Especially needed when compiling in O0. @@ -1360,6 +1350,4 @@ int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v,= intptr_t expect, #endif } =20 -#endif /* !RSEQ_SKIP_FASTPATH */ - #endif --=20 2.17.1 From nobody Tue Apr 14 22:46:25 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B9A9C00144 for ; Fri, 29 Jul 2022 19:03:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238837AbiG2TDr (ORCPT ); Fri, 29 Jul 2022 15:03:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51618 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238666AbiG2TDH (ORCPT ); Fri, 29 Jul 2022 15:03:07 -0400 Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E84E989A48; Fri, 29 Jul 2022 12:02:55 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 032D6319289; Fri, 29 Jul 2022 15:02:55 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id C_e5cOg2bAPe; Fri, 29 Jul 2022 15:02:54 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 4A9EC318AAA; Fri, 29 Jul 2022 15:02:48 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 4A9EC318AAA DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1659121368; bh=84B9YOyXv7c3EB71cteYmLNG2oHcoRDleMv6nuC252o=; h=From:To:Date:Message-Id; b=o3UsOEmjBfGQi5AiHHd+MmgnHXS8IluYh/0o7KlOtfbrUVy0zOq290Ha0tRvV/tlc /aAKOu8B2GwCfBXwh3o3hjL7/Il3ToI3tVbA8YobUZ/cvuc5qocC6s81E1bmr1mFY3 2aBDs138lLCfyusGSJNJSoVJuQiU6Ncc10mqdnNl25zy8zJt0mqvSW4IqWohHgdaF/ etE2gZ+pb/ZsGH5Q2zTOymhZljGRyKjPAXGP3Ch95JJpyIU1eKGXXJKoIcJNhvHyQg LWNVozDQSdSHeYtuVXFJh5FEbu7MMfacsZxAapsOyRVLVbfCjv2PL6yl16lSYBgOnH un9m3pAcEvDOA== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 6du7qYx850Fh; Fri, 29 Jul 2022 15:02:48 -0400 (EDT) Received: from localhost.localdomain (192-222-180-24.qc.cable.ebox.net [192.222.180.24]) by mail.efficios.com (Postfix) with ESMTPSA id 9B087318D7A; Fri, 29 Jul 2022 15:02:44 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Thomas Gleixner , "Paul E . McKenney" , Boqun Feng , "H . Peter Anvin" , Paul Turner , linux-api@vger.kernel.org, Christian Brauner , Florian Weimer , David.Laight@ACULAB.COM, carlos@redhat.com, Peter Oskolkov , Mathieu Desnoyers Subject: [PATCH v3 12/23] selftests/rseq: Implement rseq vm_vcpu_id field support Date: Fri, 29 Jul 2022 15:02:14 -0400 Message-Id: <20220729190225.12726-13-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> References: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Mathieu Desnoyers --- tools/testing/selftests/rseq/rseq-abi.h | 9 +++++++++ tools/testing/selftests/rseq/rseq.h | 10 ++++++++++ 2 files changed, 19 insertions(+) diff --git a/tools/testing/selftests/rseq/rseq-abi.h b/tools/testing/selfte= sts/rseq/rseq-abi.h index a1faa9162d52..1ee4740ebe94 100644 --- a/tools/testing/selftests/rseq/rseq-abi.h +++ b/tools/testing/selftests/rseq/rseq-abi.h @@ -155,6 +155,15 @@ struct rseq_abi { */ __u32 node_id; =20 + /* + * Restartable sequences vm_vcpu_id field. Updated by the kernel. Read by + * user-space with single-copy atomicity semantics. This field should + * only be read by the thread which registered this data structure. + * Aligned on 32-bit. Contains the current thread's virtual CPU ID + * (allocated uniquely within a memory space). + */ + __u32 vm_vcpu_id; + /* * Flexible array member at end of structure, after last feature field. */ diff --git a/tools/testing/selftests/rseq/rseq.h b/tools/testing/selftests/= rseq/rseq.h index fd17d0e54a1b..003e0e3750ce 100644 --- a/tools/testing/selftests/rseq/rseq.h +++ b/tools/testing/selftests/rseq/rseq.h @@ -191,6 +191,16 @@ static inline uint32_t rseq_current_node_id(void) return RSEQ_ACCESS_ONCE(rseq_get_abi()->node_id); } =20 +static inline bool rseq_vm_vcpu_id_available(void) +{ + return (int) rseq_feature_size >=3D rseq_offsetofend(struct rseq_abi, vm_= vcpu_id); +} + +static inline uint32_t rseq_current_vm_vcpu_id(void) +{ + return RSEQ_ACCESS_ONCE(rseq_get_abi()->vm_vcpu_id); +} + static inline void rseq_clear_rseq_cs(void) { RSEQ_WRITE_ONCE(rseq_get_abi()->rseq_cs.arch.ptr, 0); --=20 2.17.1 From nobody Tue Apr 14 22:46:25 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C7F5C04A68 for ; Fri, 29 Jul 2022 19:04:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238934AbiG2TEA (ORCPT ); Fri, 29 Jul 2022 15:04:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238720AbiG2TDm (ORCPT ); Fri, 29 Jul 2022 15:03:42 -0400 Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4DA3987363; Fri, 29 Jul 2022 12:02:58 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id E16F8319224; Fri, 29 Jul 2022 15:02:57 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id rCmgRUxmJdiD; Fri, 29 Jul 2022 15:02:55 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 151F531901E; Fri, 29 Jul 2022 15:02:51 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 151F531901E DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1659121371; bh=72wfAitEHiItqXZlrIcuqBtLmisIhXE41Z/bA9PJlsQ=; h=From:To:Date:Message-Id; b=pW2uzCNCI1hPJ9vXMYNy/sjTvBXgEa0WFeP+2Xhm6c9gYzpMoIgKlIeMOAMBMlJ4j FVrc72s8+Vx4MG1ENDJ3WI9KyF/ebviMyq8v+LFFV2dZT5gW+Z1XYGtEfjql8yucw6 /WHOClSW1bduKgysel0SyU5AGu/VCzCLrpQmSIByuzxyubdjiXNGFedKU+YzxEzGtH oqtJ0nbYqO7FpBXTfifK6bkPFpirWP5w+GfGoyIb+TbmqvFM/yGS7cJK4PTVc5NgXS YJtzo1U/XImQz6YOCuCTmeWakNa9zFW16ClZ2+eei606c6wKTaslHfiPgAq817uLIw R6aiZvu/zxjhA== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id BODA501znHfj; Fri, 29 Jul 2022 15:02:51 -0400 (EDT) Received: from localhost.localdomain (192-222-180-24.qc.cable.ebox.net [192.222.180.24]) by mail.efficios.com (Postfix) with ESMTPSA id 35BCC318A76; Fri, 29 Jul 2022 15:02:45 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Thomas Gleixner , "Paul E . McKenney" , Boqun Feng , "H . Peter Anvin" , Paul Turner , linux-api@vger.kernel.org, Christian Brauner , Florian Weimer , David.Laight@ACULAB.COM, carlos@redhat.com, Peter Oskolkov , Mathieu Desnoyers Subject: [PATCH v3 13/23] selftests/rseq: x86: Template memory ordering and percpu access mode Date: Fri, 29 Jul 2022 15:02:15 -0400 Message-Id: <20220729190225.12726-14-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> References: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Introduce a rseq-x86-bits.h template header which is internally included to generate the static inline functions covering: - relaxed and release memory ordering, - per-cpu-id and per-vm-vcpu-id per-cpu data access. This introduces changes to the rseq.h selftests API which require to update the rseq selftest programs. Similar API/templating changes need to be done for other architectures. Signed-off-by: Mathieu Desnoyers --- tools/testing/selftests/rseq/compiler.h | 6 + .../testing/selftests/rseq/rseq-bits-reset.h | 10 + .../selftests/rseq/rseq-bits-template.h | 39 + tools/testing/selftests/rseq/rseq-x86-bits.h | 993 ++++++++++++++ tools/testing/selftests/rseq/rseq-x86.h | 1181 +---------------- tools/testing/selftests/rseq/rseq.h | 159 +++ 6 files changed, 1238 insertions(+), 1150 deletions(-) create mode 100644 tools/testing/selftests/rseq/rseq-bits-reset.h create mode 100644 tools/testing/selftests/rseq/rseq-bits-template.h create mode 100644 tools/testing/selftests/rseq/rseq-x86-bits.h diff --git a/tools/testing/selftests/rseq/compiler.h b/tools/testing/selfte= sts/rseq/compiler.h index 876eb6a7f75b..f47092bddeba 100644 --- a/tools/testing/selftests/rseq/compiler.h +++ b/tools/testing/selftests/rseq/compiler.h @@ -27,4 +27,10 @@ */ #define rseq_after_asm_goto() asm volatile ("" : : : "memory") =20 +/* Combine two tokens. */ +#define RSEQ__COMBINE_TOKENS(_tokena, _tokenb) \ + _tokena##_tokenb +#define RSEQ_COMBINE_TOKENS(_tokena, _tokenb) \ + RSEQ__COMBINE_TOKENS(_tokena, _tokenb) + #endif /* RSEQ_COMPILER_H_ */ diff --git a/tools/testing/selftests/rseq/rseq-bits-reset.h b/tools/testing= /selftests/rseq/rseq-bits-reset.h new file mode 100644 index 000000000000..3016070212f3 --- /dev/null +++ b/tools/testing/selftests/rseq/rseq-bits-reset.h @@ -0,0 +1,10 @@ +/* SPDX-License-Identifier: LGPL-2.1 OR MIT */ +/* + * rseq-bits-reset.h + * + * (C) Copyright 2016-2022 - Mathieu Desnoyers + */ + +#undef RSEQ_TEMPLATE_IDENTIFIER +#undef RSEQ_TEMPLATE_CPU_ID_OFFSET +#undef RSEQ_TEMPLATE_SUFFIX diff --git a/tools/testing/selftests/rseq/rseq-bits-template.h b/tools/test= ing/selftests/rseq/rseq-bits-template.h new file mode 100644 index 000000000000..ea0fe06ef314 --- /dev/null +++ b/tools/testing/selftests/rseq/rseq-bits-template.h @@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: LGPL-2.1 OR MIT */ +/* + * rseq-bits-template.h + * + * (C) Copyright 2016-2022 - Mathieu Desnoyers + */ + +#ifdef RSEQ_TEMPLATE_CPU_ID +# define RSEQ_TEMPLATE_CPU_ID_OFFSET RSEQ_CPU_ID_OFFSET +# ifdef RSEQ_TEMPLATE_MO_RELEASE +# define RSEQ_TEMPLATE_SUFFIX _release_cpu_id +# elif defined (RSEQ_TEMPLATE_MO_RELAXED) +# define RSEQ_TEMPLATE_SUFFIX _relaxed_cpu_id +# else +# error "Never use directly; include inst= ead." +# endif +#elif defined(RSEQ_TEMPLATE_VM_VCPU_ID) +# define RSEQ_TEMPLATE_CPU_ID_OFFSET RSEQ_VM_VCPU_ID_OFFSET +# ifdef RSEQ_TEMPLATE_MO_RELEASE +# define RSEQ_TEMPLATE_SUFFIX _release_vm_vcpu_id +# elif defined (RSEQ_TEMPLATE_MO_RELAXED) +# define RSEQ_TEMPLATE_SUFFIX _relaxed_vm_vcpu_id +# else +# error "Never use directly; include inst= ead." +# endif +#elif defined (RSEQ_TEMPLATE_CPU_ID_NONE) +# ifdef RSEQ_TEMPLATE_MO_RELEASE +# define RSEQ_TEMPLATE_SUFFIX _release +# elif defined (RSEQ_TEMPLATE_MO_RELAXED) +# define RSEQ_TEMPLATE_SUFFIX _relaxed +# else +# error "Never use directly; include inst= ead." +# endif +#else +# error "Never use directly; include inste= ad." +#endif + +#define RSEQ_TEMPLATE_IDENTIFIER(x) RSEQ_COMBINE_TOKENS(x, RSEQ_TEMPLATE_S= UFFIX) + diff --git a/tools/testing/selftests/rseq/rseq-x86-bits.h b/tools/testing/s= elftests/rseq/rseq-x86-bits.h new file mode 100644 index 000000000000..28ca77cc876c --- /dev/null +++ b/tools/testing/selftests/rseq/rseq-x86-bits.h @@ -0,0 +1,993 @@ +/* SPDX-License-Identifier: LGPL-2.1 OR MIT */ +/* + * rseq-x86-bits.h + * + * (C) Copyright 2016-2022 - Mathieu Desnoyers + */ + +#include "rseq-bits-template.h" + +#ifdef __x86_64__ + +#if defined(RSEQ_TEMPLATE_MO_RELAXED) && \ + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_storev)(intptr_t *v, intptr_t exp= ect, intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), 4f) + RSEQ_INJECT_ASM(3) + "cmpq %[v], %[expect]\n\t" + "jnz %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), %l[error1]) + "cmpq %[v], %[expect]\n\t" + "jnz %l[error2]\n\t" +#endif + /* final store */ + "movq %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(5) + RSEQ_ASM_DEFINE_ABORT(4, "", abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [rseq_offset] "r" (rseq_offset), + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + : "memory", "cc", "rax" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +/* + * Compare @v against @expectnot. When it does _not_ match, load @v + * into @load, and store the content of *@v + voffp into @v. + */ +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpnev_storeoffp_load)(intptr_t *v, intp= tr_t expectnot, + long voffp, intptr_t *load, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), 4f) + RSEQ_INJECT_ASM(3) + "movq %[v], %%rbx\n\t" + "cmpq %%rbx, %[expectnot]\n\t" + "je %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), %l[error1]) + "movq %[v], %%rbx\n\t" + "cmpq %%rbx, %[expectnot]\n\t" + "je %l[error2]\n\t" +#endif + "movq %%rbx, %[load]\n\t" + "addq %[voffp], %%rbx\n\t" + "movq (%%rbx), %%rbx\n\t" + /* final store */ + "movq %%rbx, %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(5) + RSEQ_ASM_DEFINE_ABORT(4, "", abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [rseq_offset] "r" (rseq_offset), + /* final store input */ + [v] "m" (*v), + [expectnot] "r" (expectnot), + [voffp] "er" (voffp), + [load] "m" (*load) + : "memory", "cc", "rax", "rbx" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_addv)(intptr_t *v, intptr_t count, int c= pu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), 4f) + RSEQ_INJECT_ASM(3) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), %l[error1]) +#endif + /* final store */ + "addq %[count], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(4) + RSEQ_ASM_DEFINE_ABORT(4, "", abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [rseq_offset] "r" (rseq_offset), + /* final store input */ + [v] "m" (*v), + [count] "er" (count) + : "memory", "cc", "rax" + RSEQ_INJECT_CLOBBER + : abort +#ifdef RSEQ_COMPARE_TWICE + , error1 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +#endif +} + +#define RSEQ_ARCH_HAS_OFFSET_DEREF_ADDV + +/* + * pval =3D *(ptr+off) + * *pval +=3D inc; + */ +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_offset_deref_addv)(intptr_t *ptr, long o= ff, intptr_t inc, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), 4f) + RSEQ_INJECT_ASM(3) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), %l[error1]) +#endif + /* get p+v */ + "movq %[ptr], %%rbx\n\t" + "addq %[off], %%rbx\n\t" + /* get pv */ + "movq (%%rbx), %%rcx\n\t" + /* *pv +=3D inc */ + "addq %[inc], (%%rcx)\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(4) + RSEQ_ASM_DEFINE_ABORT(4, "", abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [rseq_offset] "r" (rseq_offset), + /* final store input */ + [ptr] "m" (*ptr), + [off] "er" (off), + [inc] "er" (inc) + : "memory", "cc", "rax", "rbx", "rcx" + RSEQ_INJECT_CLOBBER + : abort +#ifdef RSEQ_COMPARE_TWICE + , error1 +#endif + ); + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_cmpeqv_storev)(intptr_t *v, intpt= r_t expect, + intptr_t *v2, intptr_t expect2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), 4f) + RSEQ_INJECT_ASM(3) + "cmpq %[v], %[expect]\n\t" + "jnz %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) + "cmpq %[v2], %[expect2]\n\t" + "jnz %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), %l[error1]) + "cmpq %[v], %[expect]\n\t" + "jnz %l[error2]\n\t" + "cmpq %[v2], %[expect2]\n\t" + "jnz %l[error3]\n\t" +#endif + /* final store */ + "movq %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, "", abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [rseq_offset] "r" (rseq_offset), + /* cmp2 input */ + [v2] "m" (*v2), + [expect2] "r" (expect2), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + : "memory", "cc", "rax" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2, error3 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("1st expected value comparison failed"); +error3: + rseq_after_asm_goto(); + rseq_bug("2nd expected value comparison failed"); +#endif +} + +#endif /* #if defined(RSEQ_TEMPLATE_MO_RELAXED) && + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) */ + +#if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE= )) && \ + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trystorev_storev)(intptr_t *v, in= tptr_t expect, + intptr_t *v2, intptr_t newv2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), 4f) + RSEQ_INJECT_ASM(3) + "cmpq %[v], %[expect]\n\t" + "jnz %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), %l[error1]) + "cmpq %[v], %[expect]\n\t" + "jnz %l[error2]\n\t" +#endif + /* try store */ + "movq %[newv2], %[v2]\n\t" + RSEQ_INJECT_ASM(5) + /* final store */ + "movq %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, "", abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [rseq_offset] "r" (rseq_offset), + /* try store input */ + [v2] "m" (*v2), + [newv2] "r" (newv2), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + : "memory", "cc", "rax" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trymemcpy_storev)(intptr_t *v, in= tptr_t expect, + void *dst, void *src, size_t len, + intptr_t newv, int cpu) +{ + uint64_t rseq_scratch[3]; + + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + "movq %[src], %[rseq_scratch0]\n\t" + "movq %[dst], %[rseq_scratch1]\n\t" + "movq %[len], %[rseq_scratch2]\n\t" + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), 4f) + RSEQ_INJECT_ASM(3) + "cmpq %[v], %[expect]\n\t" + "jnz 5f\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), 6f) + "cmpq %[v], %[expect]\n\t" + "jnz 7f\n\t" +#endif + /* try memcpy */ + "test %[len], %[len]\n\t" \ + "jz 333f\n\t" \ + "222:\n\t" \ + "movb (%[src]), %%al\n\t" \ + "movb %%al, (%[dst])\n\t" \ + "inc %[src]\n\t" \ + "inc %[dst]\n\t" \ + "dec %[len]\n\t" \ + "jnz 222b\n\t" \ + "333:\n\t" \ + RSEQ_INJECT_ASM(5) + /* final store */ + "movq %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + /* teardown */ + "movq %[rseq_scratch2], %[len]\n\t" + "movq %[rseq_scratch1], %[dst]\n\t" + "movq %[rseq_scratch0], %[src]\n\t" + RSEQ_ASM_DEFINE_ABORT(4, + "movq %[rseq_scratch2], %[len]\n\t" + "movq %[rseq_scratch1], %[dst]\n\t" + "movq %[rseq_scratch0], %[src]\n\t", + abort) + RSEQ_ASM_DEFINE_CMPFAIL(5, + "movq %[rseq_scratch2], %[len]\n\t" + "movq %[rseq_scratch1], %[dst]\n\t" + "movq %[rseq_scratch0], %[src]\n\t", + cmpfail) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_CMPFAIL(6, + "movq %[rseq_scratch2], %[len]\n\t" + "movq %[rseq_scratch1], %[dst]\n\t" + "movq %[rseq_scratch0], %[src]\n\t", + error1) + RSEQ_ASM_DEFINE_CMPFAIL(7, + "movq %[rseq_scratch2], %[len]\n\t" + "movq %[rseq_scratch1], %[dst]\n\t" + "movq %[rseq_scratch0], %[src]\n\t", + error2) +#endif + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [rseq_offset] "r" (rseq_offset), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv), + /* try memcpy input */ + [dst] "r" (dst), + [src] "r" (src), + [len] "r" (len), + [rseq_scratch0] "m" (rseq_scratch[0]), + [rseq_scratch1] "m" (rseq_scratch[1]), + [rseq_scratch2] "m" (rseq_scratch[2]) + : "memory", "cc", "rax" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +#endif /* #if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_= MO_RELEASE)) && + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) */ + +#elif defined(__i386__) + +#if defined(RSEQ_TEMPLATE_MO_RELAXED) && \ + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_storev)(intptr_t *v, intptr_t exp= ect, intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), 4f) + RSEQ_INJECT_ASM(3) + "cmpl %[v], %[expect]\n\t" + "jnz %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), %l[error1]) + "cmpl %[v], %[expect]\n\t" + "jnz %l[error2]\n\t" +#endif + /* final store */ + "movl %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(5) + RSEQ_ASM_DEFINE_ABORT(4, "", abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [rseq_offset] "r" (rseq_offset), + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + : "memory", "cc", "eax" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +/* + * Compare @v against @expectnot. When it does _not_ match, load @v + * into @load, and store the content of *@v + voffp into @v. + */ +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpnev_storeoffp_load)(intptr_t *v, intp= tr_t expectnot, + long voffp, intptr_t *load, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), 4f) + RSEQ_INJECT_ASM(3) + "movl %[v], %%ebx\n\t" + "cmpl %%ebx, %[expectnot]\n\t" + "je %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), %l[error1]) + "movl %[v], %%ebx\n\t" + "cmpl %%ebx, %[expectnot]\n\t" + "je %l[error2]\n\t" +#endif + "movl %%ebx, %[load]\n\t" + "addl %[voffp], %%ebx\n\t" + "movl (%%ebx), %%ebx\n\t" + /* final store */ + "movl %%ebx, %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(5) + RSEQ_ASM_DEFINE_ABORT(4, "", abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [rseq_offset] "r" (rseq_offset), + /* final store input */ + [v] "m" (*v), + [expectnot] "r" (expectnot), + [voffp] "ir" (voffp), + [load] "m" (*load) + : "memory", "cc", "eax", "ebx" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_addv)(intptr_t *v, intptr_t count, int c= pu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), 4f) + RSEQ_INJECT_ASM(3) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), %l[error1]) +#endif + /* final store */ + "addl %[count], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(4) + RSEQ_ASM_DEFINE_ABORT(4, "", abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [rseq_offset] "r" (rseq_offset), + /* final store input */ + [v] "m" (*v), + [count] "ir" (count) + : "memory", "cc", "eax" + RSEQ_INJECT_CLOBBER + : abort +#ifdef RSEQ_COMPARE_TWICE + , error1 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_cmpeqv_storev)(intptr_t *v, intpt= r_t expect, + intptr_t *v2, intptr_t expect2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), 4f) + RSEQ_INJECT_ASM(3) + "cmpl %[v], %[expect]\n\t" + "jnz %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) + "cmpl %[expect2], %[v2]\n\t" + "jnz %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), %l[error1]) + "cmpl %[v], %[expect]\n\t" + "jnz %l[error2]\n\t" + "cmpl %[expect2], %[v2]\n\t" + "jnz %l[error3]\n\t" +#endif + "movl %[newv], %%eax\n\t" + /* final store */ + "movl %%eax, %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, "", abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [rseq_offset] "r" (rseq_offset), + /* cmp2 input */ + [v2] "m" (*v2), + [expect2] "r" (expect2), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "m" (newv) + : "memory", "cc", "eax" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2, error3 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("1st expected value comparison failed"); +error3: + rseq_after_asm_goto(); + rseq_bug("2nd expected value comparison failed"); +#endif +} + +#endif /* #if defined(RSEQ_TEMPLATE_MO_RELAXED) && + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) */ + +#if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE= )) && \ + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trystorev_storev)(intptr_t *v, in= tptr_t expect, + intptr_t *v2, intptr_t newv2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), 4f) + RSEQ_INJECT_ASM(3) + "movl %[expect], %%eax\n\t" + "cmpl %[v], %%eax\n\t" + "jnz %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), %l[error1]) + "movl %[expect], %%eax\n\t" + "cmpl %[v], %%eax\n\t" + "jnz %l[error2]\n\t" +#endif + /* try store */ + "movl %[newv2], %[v2]\n\t" + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_TEMPLATE_MO_RELEASE + "lock; addl $0,-128(%%esp)\n\t" +#endif + /* final store */ + "movl %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, "", abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [rseq_offset] "r" (rseq_offset), + /* try store input */ + [v2] "m" (*v2), + [newv2] "r" (newv2), + /* final store input */ + [v] "m" (*v), + [expect] "m" (expect), + [newv] "r" (newv) + : "memory", "cc", "eax" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif + +} + +/* TODO: implement a faster memcpy. */ +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trymemcpy_storev)(intptr_t *v, in= tptr_t expect, + void *dst, void *src, size_t len, + intptr_t newv, int cpu) +{ + uint32_t rseq_scratch[3]; + + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + "movl %[src], %[rseq_scratch0]\n\t" + "movl %[dst], %[rseq_scratch1]\n\t" + "movl %[len], %[rseq_scratch2]\n\t" + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), 4f) + RSEQ_INJECT_ASM(3) + "movl %[expect], %%eax\n\t" + "cmpl %%eax, %[v]\n\t" + "jnz 5f\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_TEMPLATE_CPU_ID_OFF= SET(%[rseq_offset]), 6f) + "movl %[expect], %%eax\n\t" + "cmpl %%eax, %[v]\n\t" + "jnz 7f\n\t" +#endif + /* try memcpy */ + "test %[len], %[len]\n\t" \ + "jz 333f\n\t" \ + "222:\n\t" \ + "movb (%[src]), %%al\n\t" \ + "movb %%al, (%[dst])\n\t" \ + "inc %[src]\n\t" \ + "inc %[dst]\n\t" \ + "dec %[len]\n\t" \ + "jnz 222b\n\t" \ + "333:\n\t" \ + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_TEMPLATE_MO_RELEASE + "lock; addl $0,-128(%%esp)\n\t" +#endif + "movl %[newv], %%eax\n\t" + /* final store */ + "movl %%eax, %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + /* teardown */ + "movl %[rseq_scratch2], %[len]\n\t" + "movl %[rseq_scratch1], %[dst]\n\t" + "movl %[rseq_scratch0], %[src]\n\t" + RSEQ_ASM_DEFINE_ABORT(4, + "movl %[rseq_scratch2], %[len]\n\t" + "movl %[rseq_scratch1], %[dst]\n\t" + "movl %[rseq_scratch0], %[src]\n\t", + abort) + RSEQ_ASM_DEFINE_CMPFAIL(5, + "movl %[rseq_scratch2], %[len]\n\t" + "movl %[rseq_scratch1], %[dst]\n\t" + "movl %[rseq_scratch0], %[src]\n\t", + cmpfail) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_CMPFAIL(6, + "movl %[rseq_scratch2], %[len]\n\t" + "movl %[rseq_scratch1], %[dst]\n\t" + "movl %[rseq_scratch0], %[src]\n\t", + error1) + RSEQ_ASM_DEFINE_CMPFAIL(7, + "movl %[rseq_scratch2], %[len]\n\t" + "movl %[rseq_scratch1], %[dst]\n\t" + "movl %[rseq_scratch0], %[src]\n\t", + error2) +#endif + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [rseq_offset] "r" (rseq_offset), + /* final store input */ + [v] "m" (*v), + [expect] "m" (expect), + [newv] "m" (newv), + /* try memcpy input */ + [dst] "r" (dst), + [src] "r" (src), + [len] "r" (len), + [rseq_scratch0] "m" (rseq_scratch[0]), + [rseq_scratch1] "m" (rseq_scratch[1]), + [rseq_scratch2] "m" (rseq_scratch[2]) + : "memory", "cc", "eax" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +#endif /* #if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_= MO_RELEASE)) && + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) */ + +#endif + +#include "rseq-bits-reset.h" diff --git a/tools/testing/selftests/rseq/rseq-x86.h b/tools/testing/selfte= sts/rseq/rseq-x86.h index e148dfb2f68a..a526a6ad3a81 100644 --- a/tools/testing/selftests/rseq/rseq-x86.h +++ b/tools/testing/selftests/rseq/rseq-x86.h @@ -2,9 +2,13 @@ /* * rseq-x86.h * - * (C) Copyright 2016-2018 - Mathieu Desnoyers + * (C) Copyright 2016-2022 - Mathieu Desnoyers */ =20 +#ifndef RSEQ_H +#error "Never use directly; include instead." +#endif + #include =20 /* @@ -22,9 +26,10 @@ * address through a "r" input operand. */ =20 -/* Offset of cpu_id and rseq_cs fields in struct rseq. */ +/* Offset of cpu_id, rseq_cs, and vm_vcpu_id fields in struct rseq. */ #define RSEQ_CPU_ID_OFFSET 4 #define RSEQ_CS_OFFSET 8 +#define RSEQ_VM_VCPU_ID_OFFSET 24 =20 #ifdef __x86_64__ =20 @@ -108,523 +113,6 @@ do { \ "jmp %l[" __rseq_str(cmpfail_label) "]\n\t" \ ".popsection\n\t" =20 -static inline __attribute__((always_inline)) -int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cp= u) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), 4f) - RSEQ_INJECT_ASM(3) - "cmpq %[v], %[expect]\n\t" - "jnz %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), %l[error1]) - "cmpq %[v], %[expect]\n\t" - "jnz %l[error2]\n\t" -#endif - /* final store */ - "movq %[newv], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(5) - RSEQ_ASM_DEFINE_ABORT(4, "", abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [rseq_offset] "r" (rseq_offset), - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - : "memory", "cc", "rax" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -/* - * Compare @v against @expectnot. When it does _not_ match, load @v - * into @load, and store the content of *@v + voffp into @v. - */ -static inline __attribute__((always_inline)) -int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, - long voffp, intptr_t *load, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), 4f) - RSEQ_INJECT_ASM(3) - "movq %[v], %%rbx\n\t" - "cmpq %%rbx, %[expectnot]\n\t" - "je %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), %l[error1]) - "movq %[v], %%rbx\n\t" - "cmpq %%rbx, %[expectnot]\n\t" - "je %l[error2]\n\t" -#endif - "movq %%rbx, %[load]\n\t" - "addq %[voffp], %%rbx\n\t" - "movq (%%rbx), %%rbx\n\t" - /* final store */ - "movq %%rbx, %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(5) - RSEQ_ASM_DEFINE_ABORT(4, "", abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [rseq_offset] "r" (rseq_offset), - /* final store input */ - [v] "m" (*v), - [expectnot] "r" (expectnot), - [voffp] "er" (voffp), - [load] "m" (*load) - : "memory", "cc", "rax", "rbx" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_addv(intptr_t *v, intptr_t count, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), 4f) - RSEQ_INJECT_ASM(3) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), %l[error1]) -#endif - /* final store */ - "addq %[count], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(4) - RSEQ_ASM_DEFINE_ABORT(4, "", abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [rseq_offset] "r" (rseq_offset), - /* final store input */ - [v] "m" (*v), - [count] "er" (count) - : "memory", "cc", "rax" - RSEQ_INJECT_CLOBBER - : abort -#ifdef RSEQ_COMPARE_TWICE - , error1 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -#endif -} - -#define RSEQ_ARCH_HAS_OFFSET_DEREF_ADDV - -/* - * pval =3D *(ptr+off) - * *pval +=3D inc; - */ -static inline __attribute__((always_inline)) -int rseq_offset_deref_addv(intptr_t *ptr, long off, intptr_t inc, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), 4f) - RSEQ_INJECT_ASM(3) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), %l[error1]) -#endif - /* get p+v */ - "movq %[ptr], %%rbx\n\t" - "addq %[off], %%rbx\n\t" - /* get pv */ - "movq (%%rbx), %%rcx\n\t" - /* *pv +=3D inc */ - "addq %[inc], (%%rcx)\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(4) - RSEQ_ASM_DEFINE_ABORT(4, "", abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [rseq_offset] "r" (rseq_offset), - /* final store input */ - [ptr] "m" (*ptr), - [off] "er" (off), - [inc] "er" (inc) - : "memory", "cc", "rax", "rbx", "rcx" - RSEQ_INJECT_CLOBBER - : abort -#ifdef RSEQ_COMPARE_TWICE - , error1 -#endif - ); - return 0; -abort: - RSEQ_INJECT_FAILED - return -1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_bug("cpu_id comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t newv2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), 4f) - RSEQ_INJECT_ASM(3) - "cmpq %[v], %[expect]\n\t" - "jnz %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), %l[error1]) - "cmpq %[v], %[expect]\n\t" - "jnz %l[error2]\n\t" -#endif - /* try store */ - "movq %[newv2], %[v2]\n\t" - RSEQ_INJECT_ASM(5) - /* final store */ - "movq %[newv], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(6) - RSEQ_ASM_DEFINE_ABORT(4, "", abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [rseq_offset] "r" (rseq_offset), - /* try store input */ - [v2] "m" (*v2), - [newv2] "r" (newv2), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - : "memory", "cc", "rax" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -/* x86-64 is TSO. */ -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t newv2, - intptr_t newv, int cpu) -{ - return rseq_cmpeqv_trystorev_storev(v, expect, v2, newv2, newv, cpu); -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t expect2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), 4f) - RSEQ_INJECT_ASM(3) - "cmpq %[v], %[expect]\n\t" - "jnz %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(4) - "cmpq %[v2], %[expect2]\n\t" - "jnz %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(5) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), %l[error1]) - "cmpq %[v], %[expect]\n\t" - "jnz %l[error2]\n\t" - "cmpq %[v2], %[expect2]\n\t" - "jnz %l[error3]\n\t" -#endif - /* final store */ - "movq %[newv], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(6) - RSEQ_ASM_DEFINE_ABORT(4, "", abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [rseq_offset] "r" (rseq_offset), - /* cmp2 input */ - [v2] "m" (*v2), - [expect2] "r" (expect2), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - : "memory", "cc", "rax" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2, error3 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("1st expected value comparison failed"); -error3: - rseq_after_asm_goto(); - rseq_bug("2nd expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, - void *dst, void *src, size_t len, - intptr_t newv, int cpu) -{ - uint64_t rseq_scratch[3]; - - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - "movq %[src], %[rseq_scratch0]\n\t" - "movq %[dst], %[rseq_scratch1]\n\t" - "movq %[len], %[rseq_scratch2]\n\t" - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), 4f) - RSEQ_INJECT_ASM(3) - "cmpq %[v], %[expect]\n\t" - "jnz 5f\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), 6f) - "cmpq %[v], %[expect]\n\t" - "jnz 7f\n\t" -#endif - /* try memcpy */ - "test %[len], %[len]\n\t" \ - "jz 333f\n\t" \ - "222:\n\t" \ - "movb (%[src]), %%al\n\t" \ - "movb %%al, (%[dst])\n\t" \ - "inc %[src]\n\t" \ - "inc %[dst]\n\t" \ - "dec %[len]\n\t" \ - "jnz 222b\n\t" \ - "333:\n\t" \ - RSEQ_INJECT_ASM(5) - /* final store */ - "movq %[newv], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(6) - /* teardown */ - "movq %[rseq_scratch2], %[len]\n\t" - "movq %[rseq_scratch1], %[dst]\n\t" - "movq %[rseq_scratch0], %[src]\n\t" - RSEQ_ASM_DEFINE_ABORT(4, - "movq %[rseq_scratch2], %[len]\n\t" - "movq %[rseq_scratch1], %[dst]\n\t" - "movq %[rseq_scratch0], %[src]\n\t", - abort) - RSEQ_ASM_DEFINE_CMPFAIL(5, - "movq %[rseq_scratch2], %[len]\n\t" - "movq %[rseq_scratch1], %[dst]\n\t" - "movq %[rseq_scratch0], %[src]\n\t", - cmpfail) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_CMPFAIL(6, - "movq %[rseq_scratch2], %[len]\n\t" - "movq %[rseq_scratch1], %[dst]\n\t" - "movq %[rseq_scratch0], %[src]\n\t", - error1) - RSEQ_ASM_DEFINE_CMPFAIL(7, - "movq %[rseq_scratch2], %[len]\n\t" - "movq %[rseq_scratch1], %[dst]\n\t" - "movq %[rseq_scratch0], %[src]\n\t", - error2) -#endif - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [rseq_offset] "r" (rseq_offset), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv), - /* try memcpy input */ - [dst] "r" (dst), - [src] "r" (src), - [len] "r" (len), - [rseq_scratch0] "m" (rseq_scratch[0]), - [rseq_scratch1] "m" (rseq_scratch[1]), - [rseq_scratch2] "m" (rseq_scratch[2]) - : "memory", "cc", "rax" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -/* x86-64 is TSO. */ -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, - void *dst, void *src, size_t len, - intptr_t newv, int cpu) -{ - return rseq_cmpeqv_trymemcpy_storev(v, expect, dst, src, len, - newv, cpu); -} - #elif defined(__i386__) =20 #define RSEQ_ASM_TP_SEGMENT %%gs @@ -711,643 +199,36 @@ do { \ "jmp %l[" __rseq_str(cmpfail_label) "]\n\t" \ ".popsection\n\t" =20 -static inline __attribute__((always_inline)) -int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cp= u) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), 4f) - RSEQ_INJECT_ASM(3) - "cmpl %[v], %[expect]\n\t" - "jnz %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), %l[error1]) - "cmpl %[v], %[expect]\n\t" - "jnz %l[error2]\n\t" -#endif - /* final store */ - "movl %[newv], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(5) - RSEQ_ASM_DEFINE_ABORT(4, "", abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [rseq_offset] "r" (rseq_offset), - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - : "memory", "cc", "eax" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -/* - * Compare @v against @expectnot. When it does _not_ match, load @v - * into @load, and store the content of *@v + voffp into @v. - */ -static inline __attribute__((always_inline)) -int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, - long voffp, intptr_t *load, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), 4f) - RSEQ_INJECT_ASM(3) - "movl %[v], %%ebx\n\t" - "cmpl %%ebx, %[expectnot]\n\t" - "je %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), %l[error1]) - "movl %[v], %%ebx\n\t" - "cmpl %%ebx, %[expectnot]\n\t" - "je %l[error2]\n\t" -#endif - "movl %%ebx, %[load]\n\t" - "addl %[voffp], %%ebx\n\t" - "movl (%%ebx), %%ebx\n\t" - /* final store */ - "movl %%ebx, %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(5) - RSEQ_ASM_DEFINE_ABORT(4, "", abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [rseq_offset] "r" (rseq_offset), - /* final store input */ - [v] "m" (*v), - [expectnot] "r" (expectnot), - [voffp] "ir" (voffp), - [load] "m" (*load) - : "memory", "cc", "eax", "ebx" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_addv(intptr_t *v, intptr_t count, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), 4f) - RSEQ_INJECT_ASM(3) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), %l[error1]) -#endif - /* final store */ - "addl %[count], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(4) - RSEQ_ASM_DEFINE_ABORT(4, "", abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [rseq_offset] "r" (rseq_offset), - /* final store input */ - [v] "m" (*v), - [count] "ir" (count) - : "memory", "cc", "eax" - RSEQ_INJECT_CLOBBER - : abort -#ifdef RSEQ_COMPARE_TWICE - , error1 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t newv2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), 4f) - RSEQ_INJECT_ASM(3) - "cmpl %[v], %[expect]\n\t" - "jnz %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), %l[error1]) - "cmpl %[v], %[expect]\n\t" - "jnz %l[error2]\n\t" -#endif - /* try store */ - "movl %[newv2], %%eax\n\t" - "movl %%eax, %[v2]\n\t" - RSEQ_INJECT_ASM(5) - /* final store */ - "movl %[newv], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(6) - RSEQ_ASM_DEFINE_ABORT(4, "", abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [rseq_offset] "r" (rseq_offset), - /* try store input */ - [v2] "m" (*v2), - [newv2] "m" (newv2), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - : "memory", "cc", "eax" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t newv2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), 4f) - RSEQ_INJECT_ASM(3) - "movl %[expect], %%eax\n\t" - "cmpl %[v], %%eax\n\t" - "jnz %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), %l[error1]) - "movl %[expect], %%eax\n\t" - "cmpl %[v], %%eax\n\t" - "jnz %l[error2]\n\t" -#endif - /* try store */ - "movl %[newv2], %[v2]\n\t" - RSEQ_INJECT_ASM(5) - "lock; addl $0,-128(%%esp)\n\t" - /* final store */ - "movl %[newv], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(6) - RSEQ_ASM_DEFINE_ABORT(4, "", abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [rseq_offset] "r" (rseq_offset), - /* try store input */ - [v2] "m" (*v2), - [newv2] "r" (newv2), - /* final store input */ - [v] "m" (*v), - [expect] "m" (expect), - [newv] "r" (newv) - : "memory", "cc", "eax" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); #endif =20 -} +/* Per-cpu-id indexing. */ =20 -static inline __attribute__((always_inline)) -int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t expect2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) +#define RSEQ_TEMPLATE_CPU_ID +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-x86-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED =20 - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), 4f) - RSEQ_INJECT_ASM(3) - "cmpl %[v], %[expect]\n\t" - "jnz %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(4) - "cmpl %[expect2], %[v2]\n\t" - "jnz %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(5) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), %l[error1]) - "cmpl %[v], %[expect]\n\t" - "jnz %l[error2]\n\t" - "cmpl %[expect2], %[v2]\n\t" - "jnz %l[error3]\n\t" -#endif - "movl %[newv], %%eax\n\t" - /* final store */ - "movl %%eax, %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(6) - RSEQ_ASM_DEFINE_ABORT(4, "", abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [rseq_offset] "r" (rseq_offset), - /* cmp2 input */ - [v2] "m" (*v2), - [expect2] "r" (expect2), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "m" (newv) - : "memory", "cc", "eax" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2, error3 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("1st expected value comparison failed"); -error3: - rseq_after_asm_goto(); - rseq_bug("2nd expected value comparison failed"); -#endif -} +#define RSEQ_TEMPLATE_MO_RELEASE +#include "rseq-x86-bits.h" +#undef RSEQ_TEMPLATE_MO_RELEASE +#undef RSEQ_TEMPLATE_CPU_ID =20 -/* TODO: implement a faster memcpy. */ -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, - void *dst, void *src, size_t len, - intptr_t newv, int cpu) -{ - uint32_t rseq_scratch[3]; +/* Per-vm-vcpu-id indexing. */ =20 - RSEQ_INJECT_C(9) +#define RSEQ_TEMPLATE_VM_VCPU_ID +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-x86-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED =20 - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - "movl %[src], %[rseq_scratch0]\n\t" - "movl %[dst], %[rseq_scratch1]\n\t" - "movl %[len], %[rseq_scratch2]\n\t" - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), 4f) - RSEQ_INJECT_ASM(3) - "movl %[expect], %%eax\n\t" - "cmpl %%eax, %[v]\n\t" - "jnz 5f\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), 6f) - "movl %[expect], %%eax\n\t" - "cmpl %%eax, %[v]\n\t" - "jnz 7f\n\t" -#endif - /* try memcpy */ - "test %[len], %[len]\n\t" \ - "jz 333f\n\t" \ - "222:\n\t" \ - "movb (%[src]), %%al\n\t" \ - "movb %%al, (%[dst])\n\t" \ - "inc %[src]\n\t" \ - "inc %[dst]\n\t" \ - "dec %[len]\n\t" \ - "jnz 222b\n\t" \ - "333:\n\t" \ - RSEQ_INJECT_ASM(5) - "movl %[newv], %%eax\n\t" - /* final store */ - "movl %%eax, %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(6) - /* teardown */ - "movl %[rseq_scratch2], %[len]\n\t" - "movl %[rseq_scratch1], %[dst]\n\t" - "movl %[rseq_scratch0], %[src]\n\t" - RSEQ_ASM_DEFINE_ABORT(4, - "movl %[rseq_scratch2], %[len]\n\t" - "movl %[rseq_scratch1], %[dst]\n\t" - "movl %[rseq_scratch0], %[src]\n\t", - abort) - RSEQ_ASM_DEFINE_CMPFAIL(5, - "movl %[rseq_scratch2], %[len]\n\t" - "movl %[rseq_scratch1], %[dst]\n\t" - "movl %[rseq_scratch0], %[src]\n\t", - cmpfail) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_CMPFAIL(6, - "movl %[rseq_scratch2], %[len]\n\t" - "movl %[rseq_scratch1], %[dst]\n\t" - "movl %[rseq_scratch0], %[src]\n\t", - error1) - RSEQ_ASM_DEFINE_CMPFAIL(7, - "movl %[rseq_scratch2], %[len]\n\t" - "movl %[rseq_scratch1], %[dst]\n\t" - "movl %[rseq_scratch0], %[src]\n\t", - error2) -#endif - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [rseq_offset] "r" (rseq_offset), - /* final store input */ - [v] "m" (*v), - [expect] "m" (expect), - [newv] "m" (newv), - /* try memcpy input */ - [dst] "r" (dst), - [src] "r" (src), - [len] "r" (len), - [rseq_scratch0] "m" (rseq_scratch[0]), - [rseq_scratch1] "m" (rseq_scratch[1]), - [rseq_scratch2] "m" (rseq_scratch[2]) - : "memory", "cc", "eax" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -/* TODO: implement a faster memcpy. */ -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, - void *dst, void *src, size_t len, - intptr_t newv, int cpu) -{ - uint32_t rseq_scratch[3]; - - RSEQ_INJECT_C(9) +#define RSEQ_TEMPLATE_MO_RELEASE +#include "rseq-x86-bits.h" +#undef RSEQ_TEMPLATE_MO_RELEASE +#undef RSEQ_TEMPLATE_VM_VCPU_ID =20 - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - "movl %[src], %[rseq_scratch0]\n\t" - "movl %[dst], %[rseq_scratch1]\n\t" - "movl %[len], %[rseq_scratch2]\n\t" - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), 4f) - RSEQ_INJECT_ASM(3) - "movl %[expect], %%eax\n\t" - "cmpl %%eax, %[v]\n\t" - "jnz 5f\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, RSEQ_ASM_TP_SEGMENT:RSEQ_CPU_ID_OFFSET(%[rse= q_offset]), 6f) - "movl %[expect], %%eax\n\t" - "cmpl %%eax, %[v]\n\t" - "jnz 7f\n\t" -#endif - /* try memcpy */ - "test %[len], %[len]\n\t" \ - "jz 333f\n\t" \ - "222:\n\t" \ - "movb (%[src]), %%al\n\t" \ - "movb %%al, (%[dst])\n\t" \ - "inc %[src]\n\t" \ - "inc %[dst]\n\t" \ - "dec %[len]\n\t" \ - "jnz 222b\n\t" \ - "333:\n\t" \ - RSEQ_INJECT_ASM(5) - "lock; addl $0,-128(%%esp)\n\t" - "movl %[newv], %%eax\n\t" - /* final store */ - "movl %%eax, %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(6) - /* teardown */ - "movl %[rseq_scratch2], %[len]\n\t" - "movl %[rseq_scratch1], %[dst]\n\t" - "movl %[rseq_scratch0], %[src]\n\t" - RSEQ_ASM_DEFINE_ABORT(4, - "movl %[rseq_scratch2], %[len]\n\t" - "movl %[rseq_scratch1], %[dst]\n\t" - "movl %[rseq_scratch0], %[src]\n\t", - abort) - RSEQ_ASM_DEFINE_CMPFAIL(5, - "movl %[rseq_scratch2], %[len]\n\t" - "movl %[rseq_scratch1], %[dst]\n\t" - "movl %[rseq_scratch0], %[src]\n\t", - cmpfail) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_CMPFAIL(6, - "movl %[rseq_scratch2], %[len]\n\t" - "movl %[rseq_scratch1], %[dst]\n\t" - "movl %[rseq_scratch0], %[src]\n\t", - error1) - RSEQ_ASM_DEFINE_CMPFAIL(7, - "movl %[rseq_scratch2], %[len]\n\t" - "movl %[rseq_scratch1], %[dst]\n\t" - "movl %[rseq_scratch0], %[src]\n\t", - error2) -#endif - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [rseq_offset] "r" (rseq_offset), - /* final store input */ - [v] "m" (*v), - [expect] "m" (expect), - [newv] "m" (newv), - /* try memcpy input */ - [dst] "r" (dst), - [src] "r" (src), - [len] "r" (len), - [rseq_scratch0] "m" (rseq_scratch[0]), - [rseq_scratch1] "m" (rseq_scratch[1]), - [rseq_scratch2] "m" (rseq_scratch[2]) - : "memory", "cc", "eax" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} +/* APIs which are not based on cpu ids. */ =20 -#endif +#define RSEQ_TEMPLATE_CPU_ID_NONE +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-x86-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED +#undef RSEQ_TEMPLATE_CPU_ID_NONE diff --git a/tools/testing/selftests/rseq/rseq.h b/tools/testing/selftests/= rseq/rseq.h index 003e0e3750ce..95a76a1c3b27 100644 --- a/tools/testing/selftests/rseq/rseq.h +++ b/tools/testing/selftests/rseq/rseq.h @@ -74,6 +74,20 @@ extern unsigned int rseq_flags; */ extern unsigned int rseq_feature_size; =20 +enum rseq_mo { + RSEQ_MO_RELAXED =3D 0, + RSEQ_MO_CONSUME =3D 1, /* Unused */ + RSEQ_MO_ACQUIRE =3D 2, /* Unused */ + RSEQ_MO_RELEASE =3D 3, + RSEQ_MO_ACQ_REL =3D 4, /* Unused */ + RSEQ_MO_SEQ_CST =3D 5, /* Unused */ +}; + +enum rseq_percpu_mode { + RSEQ_PERCPU_CPU_ID =3D 0, + RSEQ_PERCPU_VM_VCPU_ID =3D 1, +}; + static inline struct rseq_abi *rseq_get_abi(void) { return (struct rseq_abi *) ((uintptr_t) rseq_thread_pointer() + rseq_offs= et); @@ -222,4 +236,149 @@ static inline void rseq_prepare_unload(void) rseq_clear_rseq_cs(); } =20 +static inline __attribute__((always_inline)) +int rseq_cmpeqv_storev(enum rseq_mo rseq_mo, enum rseq_percpu_mode percpu_= mode, + intptr_t *v, intptr_t expect, + intptr_t newv, int cpu) +{ + if (rseq_mo !=3D RSEQ_MO_RELAXED) + return -1; + switch (percpu_mode) { + case RSEQ_PERCPU_CPU_ID: + return rseq_cmpeqv_storev_relaxed_cpu_id(v, expect, newv, cpu); + case RSEQ_PERCPU_VM_VCPU_ID: + return rseq_cmpeqv_storev_relaxed_vm_vcpu_id(v, expect, newv, cpu); + } + return -1; +} + +/* + * Compare @v against @expectnot. When it does _not_ match, load @v + * into @load, and store the content of *@v + voffp into @v. + */ +static inline __attribute__((always_inline)) +int rseq_cmpnev_storeoffp_load(enum rseq_mo rseq_mo, enum rseq_percpu_mode= percpu_mode, + intptr_t *v, intptr_t expectnot, long voffp, intptr_t *load, + int cpu) +{ + if (rseq_mo !=3D RSEQ_MO_RELAXED) + return -1; + switch (percpu_mode) { + case RSEQ_PERCPU_CPU_ID: + return rseq_cmpnev_storeoffp_load_relaxed_cpu_id(v, expectnot, voffp, lo= ad, cpu); + case RSEQ_PERCPU_VM_VCPU_ID: + return rseq_cmpnev_storeoffp_load_relaxed_vm_vcpu_id(v, expectnot, voffp= , load, cpu); + } + return -1; +} + +static inline __attribute__((always_inline)) +int rseq_addv(enum rseq_mo rseq_mo, enum rseq_percpu_mode percpu_mode, + intptr_t *v, intptr_t count, int cpu) +{ + if (rseq_mo !=3D RSEQ_MO_RELAXED) + return -1; + switch (percpu_mode) { + case RSEQ_PERCPU_CPU_ID: + return rseq_addv_relaxed_cpu_id(v, count, cpu); + case RSEQ_PERCPU_VM_VCPU_ID: + return rseq_addv_relaxed_vm_vcpu_id(v, count, cpu); + } + return -1; +} + +#ifdef RSEQ_ARCH_HAS_OFFSET_DEREF_ADDV +/* + * pval =3D *(ptr+off) + * *pval +=3D inc; + */ +static inline __attribute__((always_inline)) +int rseq_offset_deref_addv(enum rseq_mo rseq_mo, enum rseq_percpu_mode per= cpu_mode, + intptr_t *ptr, long off, intptr_t inc, int cpu) +{ + if (rseq_mo !=3D RSEQ_MO_RELAXED) + return -1; + switch (percpu_mode) { + case RSEQ_PERCPU_CPU_ID: + return rseq_offset_deref_addv_relaxed_cpu_id(ptr, off, inc, cpu); + case RSEQ_PERCPU_VM_VCPU_ID: + return rseq_offset_deref_addv_relaxed_vm_vcpu_id(ptr, off, inc, cpu); + } + return -1; +} +#endif + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trystorev_storev(enum rseq_mo rseq_mo, enum rseq_percpu_mo= de percpu_mode, + intptr_t *v, intptr_t expect, + intptr_t *v2, intptr_t newv2, + intptr_t newv, int cpu) +{ + switch (rseq_mo) { + case RSEQ_MO_RELAXED: + switch (percpu_mode) { + case RSEQ_PERCPU_CPU_ID: + return rseq_cmpeqv_trystorev_storev_relaxed_cpu_id(v, expect, v2, newv2= , newv, cpu); + case RSEQ_PERCPU_VM_VCPU_ID: + return rseq_cmpeqv_trystorev_storev_relaxed_vm_vcpu_id(v, expect, v2, n= ewv2, newv, cpu); + } + return -1; + case RSEQ_MO_RELEASE: + switch (percpu_mode) { + case RSEQ_PERCPU_CPU_ID: + return rseq_cmpeqv_trystorev_storev_release_cpu_id(v, expect, v2, newv2= , newv, cpu); + case RSEQ_PERCPU_VM_VCPU_ID: + return rseq_cmpeqv_trystorev_storev_release_vm_vcpu_id(v, expect, v2, n= ewv2, newv, cpu); + } + return -1; + default: + return -1; + } +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_cmpeqv_storev(enum rseq_mo rseq_mo, enum rseq_percpu_mode = percpu_mode, + intptr_t *v, intptr_t expect, + intptr_t *v2, intptr_t expect2, + intptr_t newv, int cpu) +{ + if (rseq_mo !=3D RSEQ_MO_RELAXED) + return -1; + switch (percpu_mode) { + case RSEQ_PERCPU_CPU_ID: + return rseq_cmpeqv_cmpeqv_storev_relaxed_cpu_id(v, expect, v2, expect2, = newv, cpu); + case RSEQ_PERCPU_VM_VCPU_ID: + return rseq_cmpeqv_cmpeqv_storev_relaxed_vm_vcpu_id(v, expect, v2, expec= t2, newv, cpu); + } + return -1; +} + +static inline __attribute__((always_inline)) +int rseq_cmpeqv_trymemcpy_storev(enum rseq_mo rseq_mo, enum rseq_percpu_mo= de percpu_mode, + intptr_t *v, intptr_t expect, + void *dst, void *src, size_t len, + intptr_t newv, int cpu) +{ + switch (rseq_mo) { + case RSEQ_MO_RELAXED: + switch (percpu_mode) { + case RSEQ_PERCPU_CPU_ID: + return rseq_cmpeqv_trymemcpy_storev_relaxed_cpu_id(v, expect, dst, src,= len, newv, cpu); + case RSEQ_PERCPU_VM_VCPU_ID: + return rseq_cmpeqv_trymemcpy_storev_relaxed_vm_vcpu_id(v, expect, dst, = src, len, newv, cpu); + } + return -1; + case RSEQ_MO_RELEASE: + switch (percpu_mode) { + case RSEQ_PERCPU_CPU_ID: + return rseq_cmpeqv_trymemcpy_storev_release_cpu_id(v, expect, dst, src,= len, newv, cpu); + case RSEQ_PERCPU_VM_VCPU_ID: + return rseq_cmpeqv_trymemcpy_storev_release_vm_vcpu_id(v, expect, dst, = src, len, newv, cpu); + } + return -1; + default: + return -1; + } +} + #endif /* RSEQ_H_ */ --=20 2.17.1 From nobody Tue Apr 14 22:46:25 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB9B0C04A68 for ; Fri, 29 Jul 2022 19:04:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238716AbiG2TEg (ORCPT ); Fri, 29 Jul 2022 15:04:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52910 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238695AbiG2TDq (ORCPT ); Fri, 29 Jul 2022 15:03:46 -0400 Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 91AB18AB36; Fri, 29 Jul 2022 12:03:05 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 9AEE7318EAC; Fri, 29 Jul 2022 15:03:02 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id TXbIU3Q_d6Wx; Fri, 29 Jul 2022 15:03:00 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 3E781319282; Fri, 29 Jul 2022 15:02:52 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 3E781319282 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1659121372; bh=xZ9VKYdxG11Igf0eolfITWkun7U+KZT60vQU2VHhu9A=; h=From:To:Date:Message-Id; b=Ca4q6ntoXQUbh0N0N3YKy4hHrexoUOpWsIto85Z22DjUzswjLqgEbjmR6C/C8ynHs jfk0CfHUujSwaYHpjM39uCRQojPp/UFlDdh+H/maH7gV4devwdZke1OgQodzyKrM2h vuUGXX3HZ1eSUMZnHVElILq6lEJk9BzXC/d5ZcKpfuJGjvWelaDp4rOCmpERpZOr0L Qs0ge/diVhhlIx3/NbbgojBqKXHuuE3sO+1wGjHJwLUYQRyZ+tEWrCtai4XeGVLaL9 eGUth0ExKNgXAOiYWKbXqAs8APvWYtZcuywzW8dKx+sjpDCgISF+4lXra2U1a5pID7 kYCKqvUbgluSg== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id O-pA7KbTPJWC; Fri, 29 Jul 2022 15:02:52 -0400 (EDT) Received: from localhost.localdomain (192-222-180-24.qc.cable.ebox.net [192.222.180.24]) by mail.efficios.com (Postfix) with ESMTPSA id 968D1318D7B; Fri, 29 Jul 2022 15:02:45 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Thomas Gleixner , "Paul E . McKenney" , Boqun Feng , "H . Peter Anvin" , Paul Turner , linux-api@vger.kernel.org, Christian Brauner , Florian Weimer , David.Laight@ACULAB.COM, carlos@redhat.com, Peter Oskolkov , Mathieu Desnoyers , Russell King , Mark Rutland Subject: [PATCH v3 14/23] selftests/rseq: arm: Template memory ordering and percpu access mode Date: Fri, 29 Jul 2022 15:02:16 -0400 Message-Id: <20220729190225.12726-15-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> References: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Introduce a rseq-arm-bits.h template header which is internally included to generate the static inline functions covering: - relaxed and release memory ordering, - per-cpu-id and per-vm-vcpu-id per-cpu data access. Signed-off-by: Mathieu Desnoyers Cc: Russell King Cc: Mark Rutland --- tools/testing/selftests/rseq/rseq-arm-bits.h | 505 ++++++++++++++ tools/testing/selftests/rseq/rseq-arm.h | 695 +------------------ 2 files changed, 530 insertions(+), 670 deletions(-) create mode 100644 tools/testing/selftests/rseq/rseq-arm-bits.h diff --git a/tools/testing/selftests/rseq/rseq-arm-bits.h b/tools/testing/s= elftests/rseq/rseq-arm-bits.h new file mode 100644 index 000000000000..60c149d40ac5 --- /dev/null +++ b/tools/testing/selftests/rseq/rseq-arm-bits.h @@ -0,0 +1,505 @@ +/* SPDX-License-Identifier: LGPL-2.1 OR MIT */ +/* + * rseq-arm-bits.h + * + * (C) Copyright 2016-2022 - Mathieu Desnoyers + */ + +#include "rseq-bits-template.h" + +#if defined(RSEQ_TEMPLATE_MO_RELAXED) && \ + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_storev)(intptr_t *v, intptr_t exp= ect, intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + "ldr r0, %[v]\n\t" + "cmp %[expect], r0\n\t" + "bne %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + "ldr r0, %[v]\n\t" + "cmp %[expect], r0\n\t" + "bne %l[error2]\n\t" +#endif + /* final store */ + "str %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(5) + "b 5f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) + "5:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "r0", "memory", "cc" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpnev_storeoffp_load)(intptr_t *v, intp= tr_t expectnot, + long voffp, intptr_t *load, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + "ldr r0, %[v]\n\t" + "cmp %[expectnot], r0\n\t" + "beq %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + "ldr r0, %[v]\n\t" + "cmp %[expectnot], r0\n\t" + "beq %l[error2]\n\t" +#endif + "str r0, %[load]\n\t" + "add r0, %[voffp]\n\t" + "ldr r0, [r0]\n\t" + /* final store */ + "str r0, %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(5) + "b 5f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) + "5:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + /* final store input */ + [v] "m" (*v), + [expectnot] "r" (expectnot), + [voffp] "Ir" (voffp), + [load] "m" (*load) + RSEQ_INJECT_INPUT + : "r0", "memory", "cc" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_addv)(intptr_t *v, intptr_t count, int c= pu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) +#endif + "ldr r0, %[v]\n\t" + "add r0, %[count]\n\t" + /* final store */ + "str r0, %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(4) + "b 5f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) + "5:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + [v] "m" (*v), + [count] "Ir" (count) + RSEQ_INJECT_INPUT + : "r0", "memory", "cc" + RSEQ_INJECT_CLOBBER + : abort +#ifdef RSEQ_COMPARE_TWICE + , error1 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_cmpeqv_storev)(intptr_t *v, intpt= r_t expect, + intptr_t *v2, intptr_t expect2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + "ldr r0, %[v]\n\t" + "cmp %[expect], r0\n\t" + "bne %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) + "ldr r0, %[v2]\n\t" + "cmp %[expect2], r0\n\t" + "bne %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + "ldr r0, %[v]\n\t" + "cmp %[expect], r0\n\t" + "bne %l[error2]\n\t" + "ldr r0, %[v2]\n\t" + "cmp %[expect2], r0\n\t" + "bne %l[error3]\n\t" +#endif + /* final store */ + "str %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + "b 5f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) + "5:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + /* cmp2 input */ + [v2] "m" (*v2), + [expect2] "r" (expect2), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "r0", "memory", "cc" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2, error3 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("1st expected value comparison failed"); +error3: + rseq_after_asm_goto(); + rseq_bug("2nd expected value comparison failed"); +#endif +} + +#endif /* #if defined(RSEQ_TEMPLATE_MO_RELAXED) && + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) */ + +#if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE= )) && \ + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trystorev_storev)(intptr_t *v, in= tptr_t expect, + intptr_t *v2, intptr_t newv2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + "ldr r0, %[v]\n\t" + "cmp %[expect], r0\n\t" + "bne %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + "ldr r0, %[v]\n\t" + "cmp %[expect], r0\n\t" + "bne %l[error2]\n\t" +#endif + /* try store */ + "str %[newv2], %[v2]\n\t" + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_TEMPLATE_MO_RELEASE + "dmb\n\t" /* full mb provides store-release */ +#endif + /* final store */ + "str %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + "b 5f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) + "5:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + /* try store input */ + [v2] "m" (*v2), + [newv2] "r" (newv2), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "r0", "memory", "cc" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trymemcpy_storev)(intptr_t *v, in= tptr_t expect, + void *dst, void *src, size_t len, + intptr_t newv, int cpu) +{ + uint32_t rseq_scratch[3]; + + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + "str %[src], %[rseq_scratch0]\n\t" + "str %[dst], %[rseq_scratch1]\n\t" + "str %[len], %[rseq_scratch2]\n\t" + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + "ldr r0, %[v]\n\t" + "cmp %[expect], r0\n\t" + "bne 5f\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 6f) + "ldr r0, %[v]\n\t" + "cmp %[expect], r0\n\t" + "bne 7f\n\t" +#endif + /* try memcpy */ + "cmp %[len], #0\n\t" \ + "beq 333f\n\t" \ + "222:\n\t" \ + "ldrb %%r0, [%[src]]\n\t" \ + "strb %%r0, [%[dst]]\n\t" \ + "adds %[src], #1\n\t" \ + "adds %[dst], #1\n\t" \ + "subs %[len], #1\n\t" \ + "bne 222b\n\t" \ + "333:\n\t" \ + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_TEMPLATE_MO_RELEASE + "dmb\n\t" /* full mb provides store-release */ +#endif + /* final store */ + "str %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + /* teardown */ + "ldr %[len], %[rseq_scratch2]\n\t" + "ldr %[dst], %[rseq_scratch1]\n\t" + "ldr %[src], %[rseq_scratch0]\n\t" + "b 8f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, + /* teardown */ + "ldr %[len], %[rseq_scratch2]\n\t" + "ldr %[dst], %[rseq_scratch1]\n\t" + "ldr %[src], %[rseq_scratch0]\n\t", + abort, 1b, 2b, 4f) + RSEQ_ASM_DEFINE_CMPFAIL(5, + /* teardown */ + "ldr %[len], %[rseq_scratch2]\n\t" + "ldr %[dst], %[rseq_scratch1]\n\t" + "ldr %[src], %[rseq_scratch0]\n\t", + cmpfail) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_CMPFAIL(6, + /* teardown */ + "ldr %[len], %[rseq_scratch2]\n\t" + "ldr %[dst], %[rseq_scratch1]\n\t" + "ldr %[src], %[rseq_scratch0]\n\t", + error1) + RSEQ_ASM_DEFINE_CMPFAIL(7, + /* teardown */ + "ldr %[len], %[rseq_scratch2]\n\t" + "ldr %[dst], %[rseq_scratch1]\n\t" + "ldr %[src], %[rseq_scratch0]\n\t", + error2) +#endif + "8:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv), + /* try memcpy input */ + [dst] "r" (dst), + [src] "r" (src), + [len] "r" (len), + [rseq_scratch0] "m" (rseq_scratch[0]), + [rseq_scratch1] "m" (rseq_scratch[1]), + [rseq_scratch2] "m" (rseq_scratch[2]) + RSEQ_INJECT_INPUT + : "r0", "memory", "cc" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +#endif /* #if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_= MO_RELEASE)) && + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) */ + +#include "rseq-bits-reset.h" diff --git a/tools/testing/selftests/rseq/rseq-arm.h b/tools/testing/selfte= sts/rseq/rseq-arm.h index 7445107f842b..eb906db604f0 100644 --- a/tools/testing/selftests/rseq/rseq-arm.h +++ b/tools/testing/selftests/rseq/rseq-arm.h @@ -2,7 +2,7 @@ /* * rseq-arm.h * - * (C) Copyright 2016-2018 - Mathieu Desnoyers + * (C) Copyright 2016-2022 - Mathieu Desnoyers */ =20 /* @@ -143,679 +143,34 @@ do { \ teardown \ "b %l[" __rseq_str(cmpfail_label) "]\n\t" =20 -static inline __attribute__((always_inline)) -int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cp= u) -{ - RSEQ_INJECT_C(9) +/* Per-cpu-id indexing. */ =20 - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - "ldr r0, %[v]\n\t" - "cmp %[expect], r0\n\t" - "bne %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - "ldr r0, %[v]\n\t" - "cmp %[expect], r0\n\t" - "bne %l[error2]\n\t" -#endif - /* final store */ - "str %[newv], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(5) - "b 5f\n\t" - RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) - "5:\n\t" - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - RSEQ_INJECT_INPUT - : "r0", "memory", "cc" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, - long voffp, intptr_t *load, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - "ldr r0, %[v]\n\t" - "cmp %[expectnot], r0\n\t" - "beq %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - "ldr r0, %[v]\n\t" - "cmp %[expectnot], r0\n\t" - "beq %l[error2]\n\t" -#endif - "str r0, %[load]\n\t" - "add r0, %[voffp]\n\t" - "ldr r0, [r0]\n\t" - /* final store */ - "str r0, %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(5) - "b 5f\n\t" - RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) - "5:\n\t" - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* final store input */ - [v] "m" (*v), - [expectnot] "r" (expectnot), - [voffp] "Ir" (voffp), - [load] "m" (*load) - RSEQ_INJECT_INPUT - : "r0", "memory", "cc" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_addv(intptr_t *v, intptr_t count, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) -#endif - "ldr r0, %[v]\n\t" - "add r0, %[count]\n\t" - /* final store */ - "str r0, %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(4) - "b 5f\n\t" - RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) - "5:\n\t" - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - [v] "m" (*v), - [count] "Ir" (count) - RSEQ_INJECT_INPUT - : "r0", "memory", "cc" - RSEQ_INJECT_CLOBBER - : abort -#ifdef RSEQ_COMPARE_TWICE - , error1 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t newv2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - "ldr r0, %[v]\n\t" - "cmp %[expect], r0\n\t" - "bne %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - "ldr r0, %[v]\n\t" - "cmp %[expect], r0\n\t" - "bne %l[error2]\n\t" -#endif - /* try store */ - "str %[newv2], %[v2]\n\t" - RSEQ_INJECT_ASM(5) - /* final store */ - "str %[newv], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(6) - "b 5f\n\t" - RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) - "5:\n\t" - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* try store input */ - [v2] "m" (*v2), - [newv2] "r" (newv2), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - RSEQ_INJECT_INPUT - : "r0", "memory", "cc" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} +#define RSEQ_TEMPLATE_CPU_ID +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-arm-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED =20 -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t newv2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) +#define RSEQ_TEMPLATE_MO_RELEASE +#include "rseq-arm-bits.h" +#undef RSEQ_TEMPLATE_MO_RELEASE +#undef RSEQ_TEMPLATE_CPU_ID =20 - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - "ldr r0, %[v]\n\t" - "cmp %[expect], r0\n\t" - "bne %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - "ldr r0, %[v]\n\t" - "cmp %[expect], r0\n\t" - "bne %l[error2]\n\t" -#endif - /* try store */ - "str %[newv2], %[v2]\n\t" - RSEQ_INJECT_ASM(5) - "dmb\n\t" /* full mb provides store-release */ - /* final store */ - "str %[newv], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(6) - "b 5f\n\t" - RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) - "5:\n\t" - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* try store input */ - [v2] "m" (*v2), - [newv2] "r" (newv2), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - RSEQ_INJECT_INPUT - : "r0", "memory", "cc" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t expect2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - "ldr r0, %[v]\n\t" - "cmp %[expect], r0\n\t" - "bne %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(4) - "ldr r0, %[v2]\n\t" - "cmp %[expect2], r0\n\t" - "bne %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(5) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - "ldr r0, %[v]\n\t" - "cmp %[expect], r0\n\t" - "bne %l[error2]\n\t" - "ldr r0, %[v2]\n\t" - "cmp %[expect2], r0\n\t" - "bne %l[error3]\n\t" -#endif - /* final store */ - "str %[newv], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(6) - "b 5f\n\t" - RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) - "5:\n\t" - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* cmp2 input */ - [v2] "m" (*v2), - [expect2] "r" (expect2), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - RSEQ_INJECT_INPUT - : "r0", "memory", "cc" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2, error3 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("1st expected value comparison failed"); -error3: - rseq_after_asm_goto(); - rseq_bug("2nd expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, - void *dst, void *src, size_t len, - intptr_t newv, int cpu) -{ - uint32_t rseq_scratch[3]; +/* Per-vm-vcpu-id indexing. */ =20 - RSEQ_INJECT_C(9) +#define RSEQ_TEMPLATE_VM_VCPU_ID +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-arm-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED =20 - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - "str %[src], %[rseq_scratch0]\n\t" - "str %[dst], %[rseq_scratch1]\n\t" - "str %[len], %[rseq_scratch2]\n\t" - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - "ldr r0, %[v]\n\t" - "cmp %[expect], r0\n\t" - "bne 5f\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 6f) - "ldr r0, %[v]\n\t" - "cmp %[expect], r0\n\t" - "bne 7f\n\t" -#endif - /* try memcpy */ - "cmp %[len], #0\n\t" \ - "beq 333f\n\t" \ - "222:\n\t" \ - "ldrb %%r0, [%[src]]\n\t" \ - "strb %%r0, [%[dst]]\n\t" \ - "adds %[src], #1\n\t" \ - "adds %[dst], #1\n\t" \ - "subs %[len], #1\n\t" \ - "bne 222b\n\t" \ - "333:\n\t" \ - RSEQ_INJECT_ASM(5) - /* final store */ - "str %[newv], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(6) - /* teardown */ - "ldr %[len], %[rseq_scratch2]\n\t" - "ldr %[dst], %[rseq_scratch1]\n\t" - "ldr %[src], %[rseq_scratch0]\n\t" - "b 8f\n\t" - RSEQ_ASM_DEFINE_ABORT(3, 4, - /* teardown */ - "ldr %[len], %[rseq_scratch2]\n\t" - "ldr %[dst], %[rseq_scratch1]\n\t" - "ldr %[src], %[rseq_scratch0]\n\t", - abort, 1b, 2b, 4f) - RSEQ_ASM_DEFINE_CMPFAIL(5, - /* teardown */ - "ldr %[len], %[rseq_scratch2]\n\t" - "ldr %[dst], %[rseq_scratch1]\n\t" - "ldr %[src], %[rseq_scratch0]\n\t", - cmpfail) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_CMPFAIL(6, - /* teardown */ - "ldr %[len], %[rseq_scratch2]\n\t" - "ldr %[dst], %[rseq_scratch1]\n\t" - "ldr %[src], %[rseq_scratch0]\n\t", - error1) - RSEQ_ASM_DEFINE_CMPFAIL(7, - /* teardown */ - "ldr %[len], %[rseq_scratch2]\n\t" - "ldr %[dst], %[rseq_scratch1]\n\t" - "ldr %[src], %[rseq_scratch0]\n\t", - error2) -#endif - "8:\n\t" - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv), - /* try memcpy input */ - [dst] "r" (dst), - [src] "r" (src), - [len] "r" (len), - [rseq_scratch0] "m" (rseq_scratch[0]), - [rseq_scratch1] "m" (rseq_scratch[1]), - [rseq_scratch2] "m" (rseq_scratch[2]) - RSEQ_INJECT_INPUT - : "r0", "memory", "cc" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, - void *dst, void *src, size_t len, - intptr_t newv, int cpu) -{ - uint32_t rseq_scratch[3]; +#define RSEQ_TEMPLATE_MO_RELEASE +#include "rseq-arm-bits.h" +#undef RSEQ_TEMPLATE_MO_RELEASE +#undef RSEQ_TEMPLATE_VM_VCPU_ID =20 - RSEQ_INJECT_C(9) +/* APIs which are not based on cpu ids. */ =20 - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - "str %[src], %[rseq_scratch0]\n\t" - "str %[dst], %[rseq_scratch1]\n\t" - "str %[len], %[rseq_scratch2]\n\t" - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - "ldr r0, %[v]\n\t" - "cmp %[expect], r0\n\t" - "bne 5f\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 6f) - "ldr r0, %[v]\n\t" - "cmp %[expect], r0\n\t" - "bne 7f\n\t" -#endif - /* try memcpy */ - "cmp %[len], #0\n\t" \ - "beq 333f\n\t" \ - "222:\n\t" \ - "ldrb %%r0, [%[src]]\n\t" \ - "strb %%r0, [%[dst]]\n\t" \ - "adds %[src], #1\n\t" \ - "adds %[dst], #1\n\t" \ - "subs %[len], #1\n\t" \ - "bne 222b\n\t" \ - "333:\n\t" \ - RSEQ_INJECT_ASM(5) - "dmb\n\t" /* full mb provides store-release */ - /* final store */ - "str %[newv], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(6) - /* teardown */ - "ldr %[len], %[rseq_scratch2]\n\t" - "ldr %[dst], %[rseq_scratch1]\n\t" - "ldr %[src], %[rseq_scratch0]\n\t" - "b 8f\n\t" - RSEQ_ASM_DEFINE_ABORT(3, 4, - /* teardown */ - "ldr %[len], %[rseq_scratch2]\n\t" - "ldr %[dst], %[rseq_scratch1]\n\t" - "ldr %[src], %[rseq_scratch0]\n\t", - abort, 1b, 2b, 4f) - RSEQ_ASM_DEFINE_CMPFAIL(5, - /* teardown */ - "ldr %[len], %[rseq_scratch2]\n\t" - "ldr %[dst], %[rseq_scratch1]\n\t" - "ldr %[src], %[rseq_scratch0]\n\t", - cmpfail) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_CMPFAIL(6, - /* teardown */ - "ldr %[len], %[rseq_scratch2]\n\t" - "ldr %[dst], %[rseq_scratch1]\n\t" - "ldr %[src], %[rseq_scratch0]\n\t", - error1) - RSEQ_ASM_DEFINE_CMPFAIL(7, - /* teardown */ - "ldr %[len], %[rseq_scratch2]\n\t" - "ldr %[dst], %[rseq_scratch1]\n\t" - "ldr %[src], %[rseq_scratch0]\n\t", - error2) -#endif - "8:\n\t" - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv), - /* try memcpy input */ - [dst] "r" (dst), - [src] "r" (src), - [len] "r" (len), - [rseq_scratch0] "m" (rseq_scratch[0]), - [rseq_scratch1] "m" (rseq_scratch[1]), - [rseq_scratch2] "m" (rseq_scratch[2]) - RSEQ_INJECT_INPUT - : "r0", "memory", "cc" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} +#define RSEQ_TEMPLATE_CPU_ID_NONE +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-arm-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED +#undef RSEQ_TEMPLATE_CPU_ID_NONE --=20 2.17.1 From nobody Tue Apr 14 22:46:25 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6F55C04A68 for ; Fri, 29 Jul 2022 19:03:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238925AbiG2TDy (ORCPT ); Fri, 29 Jul 2022 15:03:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51286 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238704AbiG2TDQ (ORCPT ); Fri, 29 Jul 2022 15:03:16 -0400 Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C0E8289EA1; Fri, 29 Jul 2022 12:02:57 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 07232318FBF; Fri, 29 Jul 2022 15:02:57 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id HbokleqE1hXu; Fri, 29 Jul 2022 15:02:55 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 40C9B318DE7; Fri, 29 Jul 2022 15:02:51 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 40C9B318DE7 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1659121371; bh=Ie7S3dyrcvpMRfxEN29qgmzq/a8aBl/lNzNhmgzwua0=; h=From:To:Date:Message-Id; b=RHGWAW+iE/7KPkx2QzLrJrDHcXAOPJkYOUwfwsXfI7eb4UvLbAu6dv5yMDbQk55he TK+X10jo+s26HrE6l4GCJfBfxwvwDHL6CMQSqYK2bKQ/uwIHGl825LcZrlZZ+yengB xv8v2XBACLHOZ71JqZzuJKA8mC+WxVBm5LAo7owleCU8NIc/I/AuKwznHjZRhjZab9 C9yCqlD4cfbA3HoSsh31EvaU0R5gcGn/qKJjPGYYnGFADNBZguSYTIMPwO+zzAPDnz AMC5Z8t1JMicOhVC4kxV64W9WLC1ZDS21zEqZN3W1DU58B5iEqryZJ/4md8mp3AuNu PZ0YzGW2Hy1xg== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 8LyJ5UvddZkm; Fri, 29 Jul 2022 15:02:51 -0400 (EDT) Received: from localhost.localdomain (192-222-180-24.qc.cable.ebox.net [192.222.180.24]) by mail.efficios.com (Postfix) with ESMTPSA id 051C4318E9A; Fri, 29 Jul 2022 15:02:46 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Thomas Gleixner , "Paul E . McKenney" , Boqun Feng , "H . Peter Anvin" , Paul Turner , linux-api@vger.kernel.org, Christian Brauner , Florian Weimer , David.Laight@ACULAB.COM, carlos@redhat.com, Peter Oskolkov , Mathieu Desnoyers , Will Deacon , Mark Rutland , Catalin Marinas Subject: [PATCH v3 15/23] selftests/rseq: arm64: Template memory ordering and percpu access mode Date: Fri, 29 Jul 2022 15:02:17 -0400 Message-Id: <20220729190225.12726-16-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> References: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Introduce a rseq-arm64-bits.h template header which is internally included to generate the static inline functions covering: - relaxed and release memory ordering, - per-cpu-id and per-vm-vcpu-id per-cpu data access. Signed-off-by: Mathieu Desnoyers Cc: Will Deacon Cc: Mark Rutland Cc: Catalin Marinas --- .../testing/selftests/rseq/rseq-arm64-bits.h | 392 +++++++++++++ tools/testing/selftests/rseq/rseq-arm64.h | 516 +----------------- 2 files changed, 422 insertions(+), 486 deletions(-) create mode 100644 tools/testing/selftests/rseq/rseq-arm64-bits.h diff --git a/tools/testing/selftests/rseq/rseq-arm64-bits.h b/tools/testing= /selftests/rseq/rseq-arm64-bits.h new file mode 100644 index 000000000000..a45a827fde44 --- /dev/null +++ b/tools/testing/selftests/rseq/rseq-arm64-bits.h @@ -0,0 +1,392 @@ +/* SPDX-License-Identifier: LGPL-2.1 OR MIT */ +/* + * rseq-arm64-bits.h + * + * (C) Copyright 2016-2022 - Mathieu Desnoyers + * (C) Copyright 2018 - Will Deacon + */ + +#include "rseq-bits-template.h" + +#if defined(RSEQ_TEMPLATE_MO_RELAXED) && \ + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_storev)(intptr_t *v, intptr_t exp= ect, intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) +#endif + RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) + RSEQ_INJECT_ASM(5) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + [v] "Qo" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpnev_storeoffp_load)(intptr_t *v, intp= tr_t expectnot, + long voffp, intptr_t *load, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPNE(v, expectnot, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + RSEQ_ASM_OP_CMPNE(v, expectnot, %l[error2]) +#endif + RSEQ_ASM_OP_R_LOAD(v) + RSEQ_ASM_OP_R_STORE(load) + RSEQ_ASM_OP_R_LOAD_OFF(voffp) + RSEQ_ASM_OP_R_FINAL_STORE(v, 3) + RSEQ_INJECT_ASM(5) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + [v] "Qo" (*v), + [expectnot] "r" (expectnot), + [load] "Qo" (*load), + [voffp] "r" (voffp) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_addv)(intptr_t *v, intptr_t count, int c= pu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) +#endif + RSEQ_ASM_OP_R_LOAD(v) + RSEQ_ASM_OP_R_ADD(count) + RSEQ_ASM_OP_R_FINAL_STORE(v, 3) + RSEQ_INJECT_ASM(4) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + [v] "Qo" (*v), + [count] "r" (count) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG + : abort +#ifdef RSEQ_COMPARE_TWICE + , error1 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_cmpeqv_storev)(intptr_t *v, intpt= r_t expect, + intptr_t *v2, intptr_t expect2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error3]) +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) + RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[cmpfail]) + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) + RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[error3]) +#endif + RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + [v] "Qo" (*v), + [expect] "r" (expect), + [v2] "Qo" (*v2), + [expect2] "r" (expect2), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2, error3 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +error3: + rseq_after_asm_goto(); + rseq_bug("2nd expected value comparison failed"); +#endif +} + +#endif /* #if defined(RSEQ_TEMPLATE_MO_RELAXED) && + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) */ + +#if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE= )) && \ + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trystorev_storev)(intptr_t *v, in= tptr_t expect, + intptr_t *v2, intptr_t newv2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) +#endif + RSEQ_ASM_OP_STORE(newv2, v2) + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_TEMPLATE_MO_RELEASE + RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3) +#else + RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) +#endif + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + [expect] "r" (expect), + [v] "Qo" (*v), + [newv] "r" (newv), + [v2] "Qo" (*v2), + [newv2] "r" (newv2) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trymemcpy_storev)(intptr_t *v, in= tptr_t expect, + void *dst, void *src, size_t len, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) +#endif + RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len) + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_TEMPLATE_MO_RELEASE + RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3) +#else + RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) +#endif + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + [expect] "r" (expect), + [v] "Qo" (*v), + [newv] "r" (newv), + [dst] "r" (dst), + [src] "r" (src), + [len] "r" (len) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG, RSEQ_ASM_TMP_REG_2 + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +#endif /* #if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_= MO_RELEASE)) && + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) */ + +#include "rseq-bits-reset.h" diff --git a/tools/testing/selftests/rseq/rseq-arm64.h b/tools/testing/self= tests/rseq/rseq-arm64.h index 49c387fcd868..7aba23cc486c 100644 --- a/tools/testing/selftests/rseq/rseq-arm64.h +++ b/tools/testing/selftests/rseq/rseq-arm64.h @@ -2,7 +2,7 @@ /* * rseq-arm64.h * - * (C) Copyright 2016-2018 - Mathieu Desnoyers + * (C) Copyright 2016-2022 - Mathieu Desnoyers * (C) Copyright 2018 - Will Deacon */ =20 @@ -200,490 +200,34 @@ do { \ " cbnz " RSEQ_ASM_TMP_REG_2 ", 222b\n" \ "333:\n" =20 -static inline __attribute__((always_inline)) -int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cp= u) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) - RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) -#endif - RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) -#endif - RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) - RSEQ_INJECT_ASM(5) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - [v] "Qo" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - RSEQ_INJECT_INPUT - : "memory", RSEQ_ASM_TMP_REG - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, - long voffp, intptr_t *load, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) - RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) -#endif - RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - RSEQ_ASM_OP_CMPNE(v, expectnot, %l[cmpfail]) - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - RSEQ_ASM_OP_CMPNE(v, expectnot, %l[error2]) -#endif - RSEQ_ASM_OP_R_LOAD(v) - RSEQ_ASM_OP_R_STORE(load) - RSEQ_ASM_OP_R_LOAD_OFF(voffp) - RSEQ_ASM_OP_R_FINAL_STORE(v, 3) - RSEQ_INJECT_ASM(5) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - [v] "Qo" (*v), - [expectnot] "r" (expectnot), - [load] "Qo" (*load), - [voffp] "r" (voffp) - RSEQ_INJECT_INPUT - : "memory", RSEQ_ASM_TMP_REG - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} +/* Per-cpu-id indexing. */ =20 -static inline __attribute__((always_inline)) -int rseq_addv(intptr_t *v, intptr_t count, int cpu) -{ - RSEQ_INJECT_C(9) +#define RSEQ_TEMPLATE_CPU_ID +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-arm64-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED =20 - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) -#endif - RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) -#endif - RSEQ_ASM_OP_R_LOAD(v) - RSEQ_ASM_OP_R_ADD(count) - RSEQ_ASM_OP_R_FINAL_STORE(v, 3) - RSEQ_INJECT_ASM(4) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - [v] "Qo" (*v), - [count] "r" (count) - RSEQ_INJECT_INPUT - : "memory", RSEQ_ASM_TMP_REG - : abort -#ifdef RSEQ_COMPARE_TWICE - , error1 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t newv2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) - RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) -#endif - RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) -#endif - RSEQ_ASM_OP_STORE(newv2, v2) - RSEQ_INJECT_ASM(5) - RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) - RSEQ_INJECT_ASM(6) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - [expect] "r" (expect), - [v] "Qo" (*v), - [newv] "r" (newv), - [v2] "Qo" (*v2), - [newv2] "r" (newv2) - RSEQ_INJECT_INPUT - : "memory", RSEQ_ASM_TMP_REG - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t newv2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) - RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) -#endif - RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) -#endif - RSEQ_ASM_OP_STORE(newv2, v2) - RSEQ_INJECT_ASM(5) - RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3) - RSEQ_INJECT_ASM(6) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - [expect] "r" (expect), - [v] "Qo" (*v), - [newv] "r" (newv), - [v2] "Qo" (*v2), - [newv2] "r" (newv2) - RSEQ_INJECT_INPUT - : "memory", RSEQ_ASM_TMP_REG - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t expect2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) - RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) - RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error3]) -#endif - RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) - RSEQ_INJECT_ASM(4) - RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[cmpfail]) - RSEQ_INJECT_ASM(5) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) - RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[error3]) -#endif - RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) - RSEQ_INJECT_ASM(6) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - [v] "Qo" (*v), - [expect] "r" (expect), - [v2] "Qo" (*v2), - [expect2] "r" (expect2), - [newv] "r" (newv) - RSEQ_INJECT_INPUT - : "memory", RSEQ_ASM_TMP_REG - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2, error3 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -error3: - rseq_after_asm_goto(); - rseq_bug("2nd expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, - void *dst, void *src, size_t len, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) - RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) -#endif - RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) -#endif - RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len) - RSEQ_INJECT_ASM(5) - RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) - RSEQ_INJECT_ASM(6) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - [expect] "r" (expect), - [v] "Qo" (*v), - [newv] "r" (newv), - [dst] "r" (dst), - [src] "r" (src), - [len] "r" (len) - RSEQ_INJECT_INPUT - : "memory", RSEQ_ASM_TMP_REG, RSEQ_ASM_TMP_REG_2 - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, - void *dst, void *src, size_t len, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) - RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(2f, %l[error2]) -#endif - RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) -#endif - RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len) - RSEQ_INJECT_ASM(5) - RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3) - RSEQ_INJECT_ASM(6) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "Qo" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - [expect] "r" (expect), - [v] "Qo" (*v), - [newv] "r" (newv), - [dst] "r" (dst), - [src] "r" (src), - [len] "r" (len) - RSEQ_INJECT_INPUT - : "memory", RSEQ_ASM_TMP_REG, RSEQ_ASM_TMP_REG_2 - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} +#define RSEQ_TEMPLATE_MO_RELEASE +#include "rseq-arm64-bits.h" +#undef RSEQ_TEMPLATE_MO_RELEASE +#undef RSEQ_TEMPLATE_CPU_ID + +/* Per-vm-vcpu-id indexing. */ + +#define RSEQ_TEMPLATE_VM_VCPU_ID +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-arm64-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED + +#define RSEQ_TEMPLATE_MO_RELEASE +#include "rseq-arm64-bits.h" +#undef RSEQ_TEMPLATE_MO_RELEASE +#undef RSEQ_TEMPLATE_VM_VCPU_ID + +/* APIs which are not based on cpu ids. */ + +#define RSEQ_TEMPLATE_CPU_ID_NONE +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-arm64-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED +#undef RSEQ_TEMPLATE_CPU_ID_NONE --=20 2.17.1 From nobody Tue Apr 14 22:46:25 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F19BC19F2A for ; Fri, 29 Jul 2022 19:03:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238786AbiG2TD5 (ORCPT ); Fri, 29 Jul 2022 15:03:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52746 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238761AbiG2TDm (ORCPT ); Fri, 29 Jul 2022 15:03:42 -0400 Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C1D5F89EBB; Fri, 29 Jul 2022 12:02:59 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id B89B5319026; Fri, 29 Jul 2022 15:02:58 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id UFC6DVVsY-YH; Fri, 29 Jul 2022 15:02:57 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 6B71D318AF2; Fri, 29 Jul 2022 15:02:51 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 6B71D318AF2 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1659121371; bh=qKFhx3jtmtKp6FvTDDSUrhUVOYJ8FTIi5369bUn879M=; h=From:To:Date:Message-Id; b=b2y3NKflzcKsvP/CPzwS8q5mf6u+ddXww6g8hPWVZEteLp0P1KMKns4N+6QKL58FP HYGgeObxQZljV/p9MAwGB3uOorm2fiyF184A8boqNnFxgVVJJyPFp97/FiSoQqyk1/ coPvVLZV9Zo7j6uClzaszFeGAcc7gf5W73fGimoF1O/Q5+f2W3qKHOdS3aOTTXqoqh JjaO48YWoFmJ2KOE/CUPwxFH7E3K39AujkQLTYmWvTcRrwW6J5VNIpBu2CGeLP2kpa CMSJhmqZeSlwHTToaKnA5P8t9eco0B0Wt78oGKGvySYOqs2tITiCsL6+B+4dTK+bQe f/VXRxCd15wVA== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 7raeHdxOdgoZ; Fri, 29 Jul 2022 15:02:51 -0400 (EDT) Received: from localhost.localdomain (192-222-180-24.qc.cable.ebox.net [192.222.180.24]) by mail.efficios.com (Postfix) with ESMTPSA id 688B5318F14; Fri, 29 Jul 2022 15:02:46 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Thomas Gleixner , "Paul E . McKenney" , Boqun Feng , "H . Peter Anvin" , Paul Turner , linux-api@vger.kernel.org, Christian Brauner , Florian Weimer , David.Laight@ACULAB.COM, carlos@redhat.com, Peter Oskolkov , Mathieu Desnoyers , Paul Burton Subject: [PATCH v3 16/23] selftests/rseq: mips: Template memory ordering and percpu access mode Date: Fri, 29 Jul 2022 15:02:18 -0400 Message-Id: <20220729190225.12726-17-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> References: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Introduce a rseq-mips-bits.h template header which is internally included to generate the static inline functions covering: - relaxed and release memory ordering, - per-cpu-id and per-vm-vcpu-id per-cpu data access. Signed-off-by: Mathieu Desnoyers Cc: Paul Burton --- tools/testing/selftests/rseq/rseq-mips-bits.h | 462 +++++++++++++ tools/testing/selftests/rseq/rseq-mips.h | 640 +----------------- 2 files changed, 487 insertions(+), 615 deletions(-) create mode 100644 tools/testing/selftests/rseq/rseq-mips-bits.h diff --git a/tools/testing/selftests/rseq/rseq-mips-bits.h b/tools/testing/= selftests/rseq/rseq-mips-bits.h new file mode 100644 index 000000000000..890e9123403f --- /dev/null +++ b/tools/testing/selftests/rseq/rseq-mips-bits.h @@ -0,0 +1,462 @@ +/* SPDX-License-Identifier: LGPL-2.1 OR MIT */ +/* + * Author: Paul Burton + * (C) Copyright 2018 MIPS Tech LLC + * (C) Copyright 2016-2022 - Mathieu Desnoyers + */ + +#include "rseq-bits-template.h" + +#if defined(RSEQ_TEMPLATE_MO_RELAXED) && \ + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_storev)(intptr_t *v, intptr_t exp= ect, intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], %l[error2]\n\t" +#endif + /* final store */ + LONG_S " %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(5) + "b 5f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) + "5:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "$4", "memory" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpnev_storeoffp_load)(intptr_t *v, intp= tr_t expectnot, + long voffp, intptr_t *load, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_L " $4, %[v]\n\t" + "beq $4, %[expectnot], %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + LONG_L " $4, %[v]\n\t" + "beq $4, %[expectnot], %l[error2]\n\t" +#endif + LONG_S " $4, %[load]\n\t" + LONG_ADDI " $4, %[voffp]\n\t" + LONG_L " $4, 0($4)\n\t" + /* final store */ + LONG_S " $4, %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(5) + "b 5f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) + "5:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + /* final store input */ + [v] "m" (*v), + [expectnot] "r" (expectnot), + [voffp] "Ir" (voffp), + [load] "m" (*load) + RSEQ_INJECT_INPUT + : "$4", "memory" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_addv)(intptr_t *v, intptr_t count, int c= pu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) +#endif + LONG_L " $4, %[v]\n\t" + LONG_ADDI " $4, %[count]\n\t" + /* final store */ + LONG_S " $4, %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(4) + "b 5f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) + "5:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + [v] "m" (*v), + [count] "Ir" (count) + RSEQ_INJECT_INPUT + : "$4", "memory" + RSEQ_INJECT_CLOBBER + : abort +#ifdef RSEQ_COMPARE_TWICE + , error1 +#endif + ); + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_cmpeqv_storev)(intptr_t *v, intpt= r_t expect, + intptr_t *v2, intptr_t expect2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) + LONG_L " $4, %[v2]\n\t" + "bne $4, %[expect2], %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], %l[error2]\n\t" + LONG_L " $4, %[v2]\n\t" + "bne $4, %[expect2], %l[error3]\n\t" +#endif + /* final store */ + LONG_S " %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + "b 5f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) + "5:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + /* cmp2 input */ + [v2] "m" (*v2), + [expect2] "r" (expect2), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "$4", "memory" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2, error3 +#endif + ); + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("1st expected value comparison failed"); +error3: + rseq_bug("2nd expected value comparison failed"); +#endif +} + +#endif /* #if defined(RSEQ_TEMPLATE_MO_RELAXED) && + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) */ + +#if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE= )) && \ + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trystorev_storev)(intptr_t *v, in= tptr_t expect, + intptr_t *v2, intptr_t newv2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], %l[error2]\n\t" +#endif + /* try store */ + LONG_S " %[newv2], %[v2]\n\t" + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_TEMPLATE_MO_RELEASE + "sync\n\t" /* full sync provides store-release */ +#endif + /* final store */ + LONG_S " %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + "b 5f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) + "5:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + /* try store input */ + [v2] "m" (*v2), + [newv2] "r" (newv2), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "$4", "memory" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trymemcpy_storev)(intptr_t *v, in= tptr_t expect, + void *dst, void *src, size_t len, + intptr_t newv, int cpu) +{ + uintptr_t rseq_scratch[3]; + + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + LONG_S " %[src], %[rseq_scratch0]\n\t" + LONG_S " %[dst], %[rseq_scratch1]\n\t" + LONG_S " %[len], %[rseq_scratch2]\n\t" + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], 5f\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 6f) + LONG_L " $4, %[v]\n\t" + "bne $4, %[expect], 7f\n\t" +#endif + /* try memcpy */ + "beqz %[len], 333f\n\t" \ + "222:\n\t" \ + "lb $4, 0(%[src])\n\t" \ + "sb $4, 0(%[dst])\n\t" \ + LONG_ADDI " %[src], 1\n\t" \ + LONG_ADDI " %[dst], 1\n\t" \ + LONG_ADDI " %[len], -1\n\t" \ + "bnez %[len], 222b\n\t" \ + "333:\n\t" \ + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_TEMPLATE_MO_RELEASE + "sync\n\t" /* full sync provides store-release */ +#endif + /* final store */ + LONG_S " %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t" + "b 8f\n\t" + RSEQ_ASM_DEFINE_ABORT(3, 4, + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + abort, 1b, 2b, 4f) + RSEQ_ASM_DEFINE_CMPFAIL(5, + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + cmpfail) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_CMPFAIL(6, + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + error1) + RSEQ_ASM_DEFINE_CMPFAIL(7, + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + error2) +#endif + "8:\n\t" + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv), + /* try memcpy input */ + [dst] "r" (dst), + [src] "r" (src), + [len] "r" (len), + [rseq_scratch0] "m" (rseq_scratch[0]), + [rseq_scratch1] "m" (rseq_scratch[1]), + [rseq_scratch2] "m" (rseq_scratch[2]) + RSEQ_INJECT_INPUT + : "$4", "memory" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +#endif /* #if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_= MO_RELEASE)) && + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) */ + +#include "rseq-bits-reset.h" diff --git a/tools/testing/selftests/rseq/rseq-mips.h b/tools/testing/selft= ests/rseq/rseq-mips.h index dd199952d649..0ca65cc088df 100644 --- a/tools/testing/selftests/rseq/rseq-mips.h +++ b/tools/testing/selftests/rseq/rseq-mips.h @@ -2,9 +2,7 @@ /* * Author: Paul Burton * (C) Copyright 2018 MIPS Tech LLC - * - * Based on rseq-arm.h: - * (C) Copyright 2016-2018 - Mathieu Desnoyers + * (C) Copyright 2016-2022 - Mathieu Desnoyers */ =20 /* @@ -150,622 +148,34 @@ do { \ teardown \ "b %l[" __rseq_str(cmpfail_label) "]\n\t" =20 -static inline __attribute__((always_inline)) -int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cp= u) -{ - RSEQ_INJECT_C(9) +/* Per-cpu-id indexing. */ =20 - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - LONG_L " $4, %[v]\n\t" - "bne $4, %[expect], %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - LONG_L " $4, %[v]\n\t" - "bne $4, %[expect], %l[error2]\n\t" -#endif - /* final store */ - LONG_S " %[newv], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(5) - "b 5f\n\t" - RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) - "5:\n\t" - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - RSEQ_INJECT_INPUT - : "$4", "memory" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - return 0; -abort: - RSEQ_INJECT_FAILED - return -1; -cmpfail: - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_bug("cpu_id comparison failed"); -error2: - rseq_bug("expected value comparison failed"); -#endif -} +#define RSEQ_TEMPLATE_CPU_ID +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-mips-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED =20 -static inline __attribute__((always_inline)) -int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, - long voffp, intptr_t *load, int cpu) -{ - RSEQ_INJECT_C(9) +#define RSEQ_TEMPLATE_MO_RELEASE +#include "rseq-mips-bits.h" +#undef RSEQ_TEMPLATE_MO_RELEASE +#undef RSEQ_TEMPLATE_CPU_ID =20 - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - LONG_L " $4, %[v]\n\t" - "beq $4, %[expectnot], %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - LONG_L " $4, %[v]\n\t" - "beq $4, %[expectnot], %l[error2]\n\t" -#endif - LONG_S " $4, %[load]\n\t" - LONG_ADDI " $4, %[voffp]\n\t" - LONG_L " $4, 0($4)\n\t" - /* final store */ - LONG_S " $4, %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(5) - "b 5f\n\t" - RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) - "5:\n\t" - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* final store input */ - [v] "m" (*v), - [expectnot] "r" (expectnot), - [voffp] "Ir" (voffp), - [load] "m" (*load) - RSEQ_INJECT_INPUT - : "$4", "memory" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - return 0; -abort: - RSEQ_INJECT_FAILED - return -1; -cmpfail: - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_bug("cpu_id comparison failed"); -error2: - rseq_bug("expected value comparison failed"); -#endif -} +/* Per-vm-vcpu-id indexing. */ =20 -static inline __attribute__((always_inline)) -int rseq_addv(intptr_t *v, intptr_t count, int cpu) -{ - RSEQ_INJECT_C(9) +#define RSEQ_TEMPLATE_VM_VCPU_ID +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-mips-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED =20 - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) -#endif - LONG_L " $4, %[v]\n\t" - LONG_ADDI " $4, %[count]\n\t" - /* final store */ - LONG_S " $4, %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(4) - "b 5f\n\t" - RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) - "5:\n\t" - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - [v] "m" (*v), - [count] "Ir" (count) - RSEQ_INJECT_INPUT - : "$4", "memory" - RSEQ_INJECT_CLOBBER - : abort -#ifdef RSEQ_COMPARE_TWICE - , error1 -#endif - ); - return 0; -abort: - RSEQ_INJECT_FAILED - return -1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_bug("cpu_id comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t newv2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) +#define RSEQ_TEMPLATE_MO_RELEASE +#include "rseq-mips-bits.h" +#undef RSEQ_TEMPLATE_MO_RELEASE +#undef RSEQ_TEMPLATE_VM_VCPU_ID =20 - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - LONG_L " $4, %[v]\n\t" - "bne $4, %[expect], %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - LONG_L " $4, %[v]\n\t" - "bne $4, %[expect], %l[error2]\n\t" -#endif - /* try store */ - LONG_S " %[newv2], %[v2]\n\t" - RSEQ_INJECT_ASM(5) - /* final store */ - LONG_S " %[newv], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(6) - "b 5f\n\t" - RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) - "5:\n\t" - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* try store input */ - [v2] "m" (*v2), - [newv2] "r" (newv2), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - RSEQ_INJECT_INPUT - : "$4", "memory" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - return 0; -abort: - RSEQ_INJECT_FAILED - return -1; -cmpfail: - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_bug("cpu_id comparison failed"); -error2: - rseq_bug("expected value comparison failed"); -#endif -} +/* APIs which are not based on cpu ids. */ =20 -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t newv2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - LONG_L " $4, %[v]\n\t" - "bne $4, %[expect], %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - LONG_L " $4, %[v]\n\t" - "bne $4, %[expect], %l[error2]\n\t" -#endif - /* try store */ - LONG_S " %[newv2], %[v2]\n\t" - RSEQ_INJECT_ASM(5) - "sync\n\t" /* full sync provides store-release */ - /* final store */ - LONG_S " %[newv], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(6) - "b 5f\n\t" - RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) - "5:\n\t" - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* try store input */ - [v2] "m" (*v2), - [newv2] "r" (newv2), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - RSEQ_INJECT_INPUT - : "$4", "memory" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - return 0; -abort: - RSEQ_INJECT_FAILED - return -1; -cmpfail: - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_bug("cpu_id comparison failed"); -error2: - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t expect2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - LONG_L " $4, %[v]\n\t" - "bne $4, %[expect], %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(4) - LONG_L " $4, %[v2]\n\t" - "bne $4, %[expect2], %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(5) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - LONG_L " $4, %[v]\n\t" - "bne $4, %[expect], %l[error2]\n\t" - LONG_L " $4, %[v2]\n\t" - "bne $4, %[expect2], %l[error3]\n\t" -#endif - /* final store */ - LONG_S " %[newv], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(6) - "b 5f\n\t" - RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f) - "5:\n\t" - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* cmp2 input */ - [v2] "m" (*v2), - [expect2] "r" (expect2), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - RSEQ_INJECT_INPUT - : "$4", "memory" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2, error3 -#endif - ); - return 0; -abort: - RSEQ_INJECT_FAILED - return -1; -cmpfail: - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_bug("cpu_id comparison failed"); -error2: - rseq_bug("1st expected value comparison failed"); -error3: - rseq_bug("2nd expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, - void *dst, void *src, size_t len, - intptr_t newv, int cpu) -{ - uintptr_t rseq_scratch[3]; - - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - LONG_S " %[src], %[rseq_scratch0]\n\t" - LONG_S " %[dst], %[rseq_scratch1]\n\t" - LONG_S " %[len], %[rseq_scratch2]\n\t" - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - LONG_L " $4, %[v]\n\t" - "bne $4, %[expect], 5f\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 6f) - LONG_L " $4, %[v]\n\t" - "bne $4, %[expect], 7f\n\t" -#endif - /* try memcpy */ - "beqz %[len], 333f\n\t" \ - "222:\n\t" \ - "lb $4, 0(%[src])\n\t" \ - "sb $4, 0(%[dst])\n\t" \ - LONG_ADDI " %[src], 1\n\t" \ - LONG_ADDI " %[dst], 1\n\t" \ - LONG_ADDI " %[len], -1\n\t" \ - "bnez %[len], 222b\n\t" \ - "333:\n\t" \ - RSEQ_INJECT_ASM(5) - /* final store */ - LONG_S " %[newv], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(6) - /* teardown */ - LONG_L " %[len], %[rseq_scratch2]\n\t" - LONG_L " %[dst], %[rseq_scratch1]\n\t" - LONG_L " %[src], %[rseq_scratch0]\n\t" - "b 8f\n\t" - RSEQ_ASM_DEFINE_ABORT(3, 4, - /* teardown */ - LONG_L " %[len], %[rseq_scratch2]\n\t" - LONG_L " %[dst], %[rseq_scratch1]\n\t" - LONG_L " %[src], %[rseq_scratch0]\n\t", - abort, 1b, 2b, 4f) - RSEQ_ASM_DEFINE_CMPFAIL(5, - /* teardown */ - LONG_L " %[len], %[rseq_scratch2]\n\t" - LONG_L " %[dst], %[rseq_scratch1]\n\t" - LONG_L " %[src], %[rseq_scratch0]\n\t", - cmpfail) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_CMPFAIL(6, - /* teardown */ - LONG_L " %[len], %[rseq_scratch2]\n\t" - LONG_L " %[dst], %[rseq_scratch1]\n\t" - LONG_L " %[src], %[rseq_scratch0]\n\t", - error1) - RSEQ_ASM_DEFINE_CMPFAIL(7, - /* teardown */ - LONG_L " %[len], %[rseq_scratch2]\n\t" - LONG_L " %[dst], %[rseq_scratch1]\n\t" - LONG_L " %[src], %[rseq_scratch0]\n\t", - error2) -#endif - "8:\n\t" - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv), - /* try memcpy input */ - [dst] "r" (dst), - [src] "r" (src), - [len] "r" (len), - [rseq_scratch0] "m" (rseq_scratch[0]), - [rseq_scratch1] "m" (rseq_scratch[1]), - [rseq_scratch2] "m" (rseq_scratch[2]) - RSEQ_INJECT_INPUT - : "$4", "memory" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - return 0; -abort: - RSEQ_INJECT_FAILED - return -1; -cmpfail: - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_bug("cpu_id comparison failed"); -error2: - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, - void *dst, void *src, size_t len, - intptr_t newv, int cpu) -{ - uintptr_t rseq_scratch[3]; - - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(9, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - LONG_S " %[src], %[rseq_scratch0]\n\t" - LONG_S " %[dst], %[rseq_scratch1]\n\t" - LONG_S " %[len], %[rseq_scratch2]\n\t" - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - LONG_L " $4, %[v]\n\t" - "bne $4, %[expect], 5f\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 6f) - LONG_L " $4, %[v]\n\t" - "bne $4, %[expect], 7f\n\t" -#endif - /* try memcpy */ - "beqz %[len], 333f\n\t" \ - "222:\n\t" \ - "lb $4, 0(%[src])\n\t" \ - "sb $4, 0(%[dst])\n\t" \ - LONG_ADDI " %[src], 1\n\t" \ - LONG_ADDI " %[dst], 1\n\t" \ - LONG_ADDI " %[len], -1\n\t" \ - "bnez %[len], 222b\n\t" \ - "333:\n\t" \ - RSEQ_INJECT_ASM(5) - "sync\n\t" /* full sync provides store-release */ - /* final store */ - LONG_S " %[newv], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(6) - /* teardown */ - LONG_L " %[len], %[rseq_scratch2]\n\t" - LONG_L " %[dst], %[rseq_scratch1]\n\t" - LONG_L " %[src], %[rseq_scratch0]\n\t" - "b 8f\n\t" - RSEQ_ASM_DEFINE_ABORT(3, 4, - /* teardown */ - LONG_L " %[len], %[rseq_scratch2]\n\t" - LONG_L " %[dst], %[rseq_scratch1]\n\t" - LONG_L " %[src], %[rseq_scratch0]\n\t", - abort, 1b, 2b, 4f) - RSEQ_ASM_DEFINE_CMPFAIL(5, - /* teardown */ - LONG_L " %[len], %[rseq_scratch2]\n\t" - LONG_L " %[dst], %[rseq_scratch1]\n\t" - LONG_L " %[src], %[rseq_scratch0]\n\t", - cmpfail) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_CMPFAIL(6, - /* teardown */ - LONG_L " %[len], %[rseq_scratch2]\n\t" - LONG_L " %[dst], %[rseq_scratch1]\n\t" - LONG_L " %[src], %[rseq_scratch0]\n\t", - error1) - RSEQ_ASM_DEFINE_CMPFAIL(7, - /* teardown */ - LONG_L " %[len], %[rseq_scratch2]\n\t" - LONG_L " %[dst], %[rseq_scratch1]\n\t" - LONG_L " %[src], %[rseq_scratch0]\n\t", - error2) -#endif - "8:\n\t" - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv), - /* try memcpy input */ - [dst] "r" (dst), - [src] "r" (src), - [len] "r" (len), - [rseq_scratch0] "m" (rseq_scratch[0]), - [rseq_scratch1] "m" (rseq_scratch[1]), - [rseq_scratch2] "m" (rseq_scratch[2]) - RSEQ_INJECT_INPUT - : "$4", "memory" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - return 0; -abort: - RSEQ_INJECT_FAILED - return -1; -cmpfail: - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_bug("cpu_id comparison failed"); -error2: - rseq_bug("expected value comparison failed"); -#endif -} +#define RSEQ_TEMPLATE_CPU_ID_NONE +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-mips-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED +#undef RSEQ_TEMPLATE_CPU_ID_NONE --=20 2.17.1 From nobody Tue Apr 14 22:46:25 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB98BC00144 for ; Fri, 29 Jul 2022 19:04:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238742AbiG2TED (ORCPT ); Fri, 29 Jul 2022 15:04:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52770 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238775AbiG2TDn (ORCPT ); Fri, 29 Jul 2022 15:03:43 -0400 Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7687788E39; Fri, 29 Jul 2022 12:03:00 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 83B4D318F2B; Fri, 29 Jul 2022 15:02:59 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 4Cbv7mmQqxI8; Fri, 29 Jul 2022 15:02:57 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 999C1318B61; Fri, 29 Jul 2022 15:02:51 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 999C1318B61 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1659121371; bh=bzVOti+NSNFdOqSVptn18jajeCKw16PqsoJT6G0pVXc=; h=From:To:Date:Message-Id; b=rsBX12EGz8O9Ovw7TZwV/Q20LQUsyuH6faDIkm22SHo+abDSiTu7nyMI6t5SpkpLZ e1Qvx5hVg5U7/l8iqFouw94QYLULLin6RIDTlwnAlZbKafsfB61jkBX29PODdGVYoD pI+3Slh3kQjOoFOw/1bAXrU/A+luBNpujy9ALa4iqf4pAPfPh0tCP0+LHg4Pikabdt 7nJhosjpu/YnKypFfntoM6dK/rz2c5kHogNpzYjuxAAVQYgr73KZO51fDPNG4fP4kM rEQ4c8/kVIuukhILZ0cRfcfrsPwMnBjR1idxjkFu+Ci6JRtAVHPSPdx/iXaJyAwPNR 5UagL52rqNHjA== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id JrX13Pe2ARu5; Fri, 29 Jul 2022 15:02:51 -0400 (EDT) Received: from localhost.localdomain (192-222-180-24.qc.cable.ebox.net [192.222.180.24]) by mail.efficios.com (Postfix) with ESMTPSA id BE442318B55; Fri, 29 Jul 2022 15:02:46 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Thomas Gleixner , "Paul E . McKenney" , Boqun Feng , "H . Peter Anvin" , Paul Turner , linux-api@vger.kernel.org, Christian Brauner , Florian Weimer , David.Laight@ACULAB.COM, carlos@redhat.com, Peter Oskolkov , Mathieu Desnoyers Subject: [PATCH v3 17/23] selftests/rseq: ppc: Template memory ordering and percpu access mode Date: Fri, 29 Jul 2022 15:02:19 -0400 Message-Id: <20220729190225.12726-18-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> References: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Introduce a rseq-ppc-bits.h template header which is internally included to generate the static inline functions covering: - relaxed and release memory ordering, - per-cpu-id and per-vm-vcpu-id per-cpu data access. Signed-off-by: Mathieu Desnoyers Cc: Boqun Feng --- tools/testing/selftests/rseq/rseq-ppc-bits.h | 454 ++++++++++++++ tools/testing/selftests/rseq/rseq-ppc.h | 611 +------------------ 2 files changed, 486 insertions(+), 579 deletions(-) create mode 100644 tools/testing/selftests/rseq/rseq-ppc-bits.h diff --git a/tools/testing/selftests/rseq/rseq-ppc-bits.h b/tools/testing/s= elftests/rseq/rseq-ppc-bits.h new file mode 100644 index 000000000000..2d33c97c9236 --- /dev/null +++ b/tools/testing/selftests/rseq/rseq-ppc-bits.h @@ -0,0 +1,454 @@ +/* SPDX-License-Identifier: LGPL-2.1 OR MIT */ +/* + * rseq-ppc-bits.h + * + * (C) Copyright 2016-2018 - Mathieu Desnoyers + * (C) Copyright 2016-2018 - Boqun Feng + */ + +#include "rseq-bits-template.h" + +#if defined(RSEQ_TEMPLATE_MO_RELAXED) && \ + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_storev)(intptr_t *v, intptr_t exp= ect, intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + /* cmp @v equal to @expect */ + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + /* cmp @v equal to @expect */ + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) +#endif + /* final store */ + RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) + RSEQ_INJECT_ASM(5) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "memory", "cc", "r17" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpnev_storeoffp_load)(intptr_t *v, intp= tr_t expectnot, + long voffp, intptr_t *load, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + /* cmp @v not equal to @expectnot */ + RSEQ_ASM_OP_CMPNE(v, expectnot, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + /* cmp @v not equal to @expectnot */ + RSEQ_ASM_OP_CMPNE(v, expectnot, %l[error2]) +#endif + /* load the value of @v */ + RSEQ_ASM_OP_R_LOAD(v) + /* store it in @load */ + RSEQ_ASM_OP_R_STORE(load) + /* dereference voffp(v) */ + RSEQ_ASM_OP_R_LOADX(voffp) + /* final store the value at voffp(v) */ + RSEQ_ASM_OP_R_FINAL_STORE(v, 2) + RSEQ_INJECT_ASM(5) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + /* final store input */ + [v] "m" (*v), + [expectnot] "r" (expectnot), + [voffp] "b" (voffp), + [load] "m" (*load) + RSEQ_INJECT_INPUT + : "memory", "cc", "r17" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_addv)(intptr_t *v, intptr_t count, int c= pu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) +#ifdef RSEQ_COMPARE_TWICE + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) +#endif + /* load the value of @v */ + RSEQ_ASM_OP_R_LOAD(v) + /* add @count to it */ + RSEQ_ASM_OP_R_ADD(count) + /* final store */ + RSEQ_ASM_OP_R_FINAL_STORE(v, 2) + RSEQ_INJECT_ASM(4) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + /* final store input */ + [v] "m" (*v), + [count] "r" (count) + RSEQ_INJECT_INPUT + : "memory", "cc", "r17" + RSEQ_INJECT_CLOBBER + : abort +#ifdef RSEQ_COMPARE_TWICE + , error1 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_cmpeqv_storev)(intptr_t *v, intpt= r_t expect, + intptr_t *v2, intptr_t expect2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + /* cmp @v equal to @expect */ + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) + /* cmp @v2 equal to @expct2 */ + RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[cmpfail]) + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_COMPARE_TWICE + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + /* cmp @v equal to @expect */ + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) + /* cmp @v2 equal to @expct2 */ + RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[error3]) +#endif + /* final store */ + RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + /* cmp2 input */ + [v2] "m" (*v2), + [expect2] "r" (expect2), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "memory", "cc", "r17" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2, error3 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("1st expected value comparison failed"); +error3: + rseq_after_asm_goto(); + rseq_bug("2nd expected value comparison failed"); +#endif +} + +#endif /* #if defined(RSEQ_TEMPLATE_MO_RELAXED) && + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) */ + +#if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE= )) && \ + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trystorev_storev)(intptr_t *v, in= tptr_t expect, + intptr_t *v2, intptr_t newv2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + /* cmp @v equal to @expect */ + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + /* cmp @v equal to @expect */ + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) +#endif + /* try store */ + RSEQ_ASM_OP_STORE(newv2, v2) + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_TEMPLATE_MO_RELEASE + /* for 'release' */ + "lwsync\n\t" +#endif + /* final store */ + RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + /* try store input */ + [v2] "m" (*v2), + [newv2] "r" (newv2), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "memory", "cc", "r17" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trymemcpy_storev)(intptr_t *v, in= tptr_t expect, + void *dst, void *src, size_t len, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + /* setup for mempcy */ + "mr %%r19, %[len]\n\t" + "mr %%r20, %[src]\n\t" + "mr %%r21, %[dst]\n\t" + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + /* cmp @v equal to @expect */ + RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + /* cmp cpuid */ + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + /* cmp @v equal to @expect */ + RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) +#endif + /* try memcpy */ + RSEQ_ASM_OP_R_MEMCPY() + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_TEMPLATE_MO_RELEASE + /* for 'release' */ + "lwsync\n\t" +#endif + /* final store */ + RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) + RSEQ_INJECT_ASM(6) + /* teardown */ + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv), + /* try memcpy input */ + [dst] "r" (dst), + [src] "r" (src), + [len] "r" (len) + RSEQ_INJECT_INPUT + : "memory", "cc", "r17", "r18", "r19", "r20", "r21" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +#endif /* #if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_= MO_RELEASE)) && + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) */ + +#include "rseq-bits-reset.h" diff --git a/tools/testing/selftests/rseq/rseq-ppc.h b/tools/testing/selfte= sts/rseq/rseq-ppc.h index f82d95c1bb3f..78015fc52e72 100644 --- a/tools/testing/selftests/rseq/rseq-ppc.h +++ b/tools/testing/selftests/rseq/rseq-ppc.h @@ -2,7 +2,7 @@ /* * rseq-ppc.h * - * (C) Copyright 2016-2018 - Mathieu Desnoyers + * (C) Copyright 2016-2022 - Mathieu Desnoyers * (C) Copyright 2016-2018 - Boqun Feng */ =20 @@ -205,581 +205,34 @@ do { \ RSEQ_STORE_LONG(var) "%[" __rseq_str(value) "], %[" __rseq_str(var) "]\n= \t" \ __rseq_str(post_commit_label) ":\n\t" =20 -static inline __attribute__((always_inline)) -int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cp= u) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) -#endif - /* final store */ - RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) - RSEQ_INJECT_ASM(5) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - RSEQ_INJECT_INPUT - : "memory", "cc", "r17" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, - long voffp, intptr_t *load, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - /* cmp @v not equal to @expectnot */ - RSEQ_ASM_OP_CMPNE(v, expectnot, %l[cmpfail]) - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - /* cmp @v not equal to @expectnot */ - RSEQ_ASM_OP_CMPNE(v, expectnot, %l[error2]) -#endif - /* load the value of @v */ - RSEQ_ASM_OP_R_LOAD(v) - /* store it in @load */ - RSEQ_ASM_OP_R_STORE(load) - /* dereference voffp(v) */ - RSEQ_ASM_OP_R_LOADX(voffp) - /* final store the value at voffp(v) */ - RSEQ_ASM_OP_R_FINAL_STORE(v, 2) - RSEQ_INJECT_ASM(5) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* final store input */ - [v] "m" (*v), - [expectnot] "r" (expectnot), - [voffp] "b" (voffp), - [load] "m" (*load) - RSEQ_INJECT_INPUT - : "memory", "cc", "r17" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_addv(intptr_t *v, intptr_t count, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) -#ifdef RSEQ_COMPARE_TWICE - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) -#endif - /* load the value of @v */ - RSEQ_ASM_OP_R_LOAD(v) - /* add @count to it */ - RSEQ_ASM_OP_R_ADD(count) - /* final store */ - RSEQ_ASM_OP_R_FINAL_STORE(v, 2) - RSEQ_INJECT_ASM(4) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* final store input */ - [v] "m" (*v), - [count] "r" (count) - RSEQ_INJECT_INPUT - : "memory", "cc", "r17" - RSEQ_INJECT_CLOBBER - : abort -#ifdef RSEQ_COMPARE_TWICE - , error1 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t newv2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) -#endif - /* try store */ - RSEQ_ASM_OP_STORE(newv2, v2) - RSEQ_INJECT_ASM(5) - /* final store */ - RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) - RSEQ_INJECT_ASM(6) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* try store input */ - [v2] "m" (*v2), - [newv2] "r" (newv2), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - RSEQ_INJECT_INPUT - : "memory", "cc", "r17" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t newv2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) -#endif - /* try store */ - RSEQ_ASM_OP_STORE(newv2, v2) - RSEQ_INJECT_ASM(5) - /* for 'release' */ - "lwsync\n\t" - /* final store */ - RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) - RSEQ_INJECT_ASM(6) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* try store input */ - [v2] "m" (*v2), - [newv2] "r" (newv2), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - RSEQ_INJECT_INPUT - : "memory", "cc", "r17" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t expect2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) - RSEQ_INJECT_ASM(4) - /* cmp @v2 equal to @expct2 */ - RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[cmpfail]) - RSEQ_INJECT_ASM(5) -#ifdef RSEQ_COMPARE_TWICE - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) - /* cmp @v2 equal to @expct2 */ - RSEQ_ASM_OP_CMPEQ(v2, expect2, %l[error3]) -#endif - /* final store */ - RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) - RSEQ_INJECT_ASM(6) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* cmp2 input */ - [v2] "m" (*v2), - [expect2] "r" (expect2), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - RSEQ_INJECT_INPUT - : "memory", "cc", "r17" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2, error3 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("1st expected value comparison failed"); -error3: - rseq_after_asm_goto(); - rseq_bug("2nd expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, - void *dst, void *src, size_t len, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* setup for mempcy */ - "mr %%r19, %[len]\n\t" - "mr %%r20, %[src]\n\t" - "mr %%r21, %[dst]\n\t" - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) -#endif - /* try memcpy */ - RSEQ_ASM_OP_R_MEMCPY() - RSEQ_INJECT_ASM(5) - /* final store */ - RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) - RSEQ_INJECT_ASM(6) - /* teardown */ - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv), - /* try memcpy input */ - [dst] "r" (dst), - [src] "r" (src), - [len] "r" (len) - RSEQ_INJECT_INPUT - : "memory", "cc", "r17", "r18", "r19", "r20", "r21" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, - void *dst, void *src, size_t len, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* setup for mempcy */ - "mr %%r19, %[len]\n\t" - "mr %%r20, %[src]\n\t" - "mr %%r21, %[dst]\n\t" - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[cmpfail]) - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - /* cmp cpuid */ - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - /* cmp @v equal to @expect */ - RSEQ_ASM_OP_CMPEQ(v, expect, %l[error2]) -#endif - /* try memcpy */ - RSEQ_ASM_OP_R_MEMCPY() - RSEQ_INJECT_ASM(5) - /* for 'release' */ - "lwsync\n\t" - /* final store */ - RSEQ_ASM_OP_FINAL_STORE(newv, v, 2) - RSEQ_INJECT_ASM(6) - /* teardown */ - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv), - /* try memcpy input */ - [dst] "r" (dst), - [src] "r" (src), - [len] "r" (len) - RSEQ_INJECT_INPUT - : "memory", "cc", "r17", "r18", "r19", "r20", "r21" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} +/* Per-cpu-id indexing. */ + +#define RSEQ_TEMPLATE_CPU_ID +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-ppc-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED + +#define RSEQ_TEMPLATE_MO_RELEASE +#include "rseq-ppc-bits.h" +#undef RSEQ_TEMPLATE_MO_RELEASE +#undef RSEQ_TEMPLATE_CPU_ID + +/* Per-vm-vcpu-id indexing. */ + +#define RSEQ_TEMPLATE_VM_VCPU_ID +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-ppc-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED + +#define RSEQ_TEMPLATE_MO_RELEASE +#include "rseq-ppc-bits.h" +#undef RSEQ_TEMPLATE_MO_RELEASE +#undef RSEQ_TEMPLATE_VM_VCPU_ID + +/* APIs which are not based on cpu ids. */ + +#define RSEQ_TEMPLATE_CPU_ID_NONE +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-ppc-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED +#undef RSEQ_TEMPLATE_CPU_ID_NONE --=20 2.17.1 From nobody Tue Apr 14 22:46:25 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3B16C00144 for ; Fri, 29 Jul 2022 19:04:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238957AbiG2TEI (ORCPT ); Fri, 29 Jul 2022 15:04:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238787AbiG2TDn (ORCPT ); Fri, 29 Jul 2022 15:03:43 -0400 Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 72AF08AB0A; Fri, 29 Jul 2022 12:03:01 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 385C43190A0; Fri, 29 Jul 2022 15:03:00 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id cuG6r6BMcDnd; Fri, 29 Jul 2022 15:02:58 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id C3F44319181; Fri, 29 Jul 2022 15:02:51 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com C3F44319181 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1659121371; bh=Z3yeGxlT4nwKTt7T64urS3ecuU+bpn3abP7/tXvN38U=; h=From:To:Date:Message-Id; b=kUIfSNSHQO0kQyeGGoFU1zpj83qyqD9cXtOr6TCAJVPXi4ZSjPWRGrafEvsnFFgpu rdUKuF37Ct4Lu9aMRS5WVk2v9mnR+972sB1TO+U9IP0MzNlCT9+Ldcmzw2newNHo5N vFwIFt66qtkLTizpAyL3Z2IKudUQMQqFrh68ilczruAZfRgUZXIsQhBOTj3fHCMc4R +3Xnj118fGYv1qvDnt7Zz0mR0yAdz4GBJxmEN2CFwzRvD5r6SlIbd6trDF0RF2ybzj KGvTMF42HEeaksL1QGXvrKF+wZ3eC2hwKpq1lBiX0l4/6ijJZqPSEdB0o2LbVIOFJe jBmWJE+Tdh6gg== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id Q4Vc-CK68t5Q; Fri, 29 Jul 2022 15:02:51 -0400 (EDT) Received: from localhost.localdomain (192-222-180-24.qc.cable.ebox.net [192.222.180.24]) by mail.efficios.com (Postfix) with ESMTPSA id 20574318C73; Fri, 29 Jul 2022 15:02:47 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Thomas Gleixner , "Paul E . McKenney" , Boqun Feng , "H . Peter Anvin" , Paul Turner , linux-api@vger.kernel.org, Christian Brauner , Florian Weimer , David.Laight@ACULAB.COM, carlos@redhat.com, Peter Oskolkov , Mathieu Desnoyers , Vasily Gorbik , Heiko Carstens , Martin Schwidefsky Subject: [PATCH v3 18/23] selftests/rseq: s390: Template memory ordering and percpu access mode Date: Fri, 29 Jul 2022 15:02:20 -0400 Message-Id: <20220729190225.12726-19-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> References: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Introduce a rseq-s390-bits.h template header which is internally included to generate the static inline functions covering: - relaxed and release memory ordering, - per-cpu-id and per-vm-vcpu-id per-cpu data access. Signed-off-by: Mathieu Desnoyers Cc: Vasily Gorbik Cc: Heiko Carstens Cc: Martin Schwidefsky --- tools/testing/selftests/rseq/rseq-s390-bits.h | 474 +++++++++++++++++ tools/testing/selftests/rseq/rseq-s390.h | 490 +----------------- 2 files changed, 498 insertions(+), 466 deletions(-) create mode 100644 tools/testing/selftests/rseq/rseq-s390-bits.h diff --git a/tools/testing/selftests/rseq/rseq-s390-bits.h b/tools/testing/= selftests/rseq/rseq-s390-bits.h new file mode 100644 index 000000000000..78a53e6aeda1 --- /dev/null +++ b/tools/testing/selftests/rseq/rseq-s390-bits.h @@ -0,0 +1,474 @@ +/* SPDX-License-Identifier: LGPL-2.1 OR MIT */ + +#include "rseq-bits-template.h" + +#if defined(RSEQ_TEMPLATE_MO_RELAXED) && \ + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_storev)(intptr_t *v, intptr_t exp= ect, intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_CMP " %[expect], %[v]\n\t" + "jnz %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + LONG_CMP " %[expect], %[v]\n\t" + "jnz %l[error2]\n\t" +#endif + /* final store */ + LONG_S " %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(5) + RSEQ_ASM_DEFINE_ABORT(4, "", abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "memory", "cc", "r0" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +/* + * Compare @v against @expectnot. When it does _not_ match, load @v + * into @load, and store the content of *@v + voffp into @v. + */ +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpnev_storeoffp_load)(intptr_t *v, intp= tr_t expectnot, + long voffp, intptr_t *load, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_L " %%r1, %[v]\n\t" + LONG_CMP_R " %%r1, %[expectnot]\n\t" + "je %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + LONG_L " %%r1, %[v]\n\t" + LONG_CMP_R " %%r1, %[expectnot]\n\t" + "je %l[error2]\n\t" +#endif + LONG_S " %%r1, %[load]\n\t" + LONG_ADD_R " %%r1, %[voffp]\n\t" + LONG_L " %%r1, 0(%%r1)\n\t" + /* final store */ + LONG_S " %%r1, %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(5) + RSEQ_ASM_DEFINE_ABORT(4, "", abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + /* final store input */ + [v] "m" (*v), + [expectnot] "r" (expectnot), + [voffp] "r" (voffp), + [load] "m" (*load) + RSEQ_INJECT_INPUT + : "memory", "cc", "r0", "r1" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_addv)(intptr_t *v, intptr_t count, int c= pu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) +#endif + LONG_L " %%r0, %[v]\n\t" + LONG_ADD_R " %%r0, %[count]\n\t" + /* final store */ + LONG_S " %%r0, %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(4) + RSEQ_ASM_DEFINE_ABORT(4, "", abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + /* final store input */ + [v] "m" (*v), + [count] "r" (count) + RSEQ_INJECT_INPUT + : "memory", "cc", "r0" + RSEQ_INJECT_CLOBBER + : abort +#ifdef RSEQ_COMPARE_TWICE + , error1 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +#endif +} + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_cmpeqv_storev)(intptr_t *v, intpt= r_t expect, + intptr_t *v2, intptr_t expect2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_CMP " %[expect], %[v]\n\t" + "jnz %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) + LONG_CMP " %[expect2], %[v2]\n\t" + "jnz %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + LONG_CMP " %[expect], %[v]\n\t" + "jnz %l[error2]\n\t" + LONG_CMP " %[expect2], %[v2]\n\t" + "jnz %l[error3]\n\t" +#endif + /* final store */ + LONG_S " %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, "", abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + /* cmp2 input */ + [v2] "m" (*v2), + [expect2] "r" (expect2), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "memory", "cc", "r0" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2, error3 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("1st expected value comparison failed"); +error3: + rseq_after_asm_goto(); + rseq_bug("2nd expected value comparison failed"); +#endif +} + +#endif /* #if defined(RSEQ_TEMPLATE_MO_RELAXED) && + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) */ + +#if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE= )) && \ + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) + +/* s390 is TSO. */ +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trystorev_storev)(intptr_t *v, in= tptr_t expect, + intptr_t *v2, intptr_t newv2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_CMP " %[expect], %[v]\n\t" + "jnz %l[cmpfail]\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) + LONG_CMP " %[expect], %[v]\n\t" + "jnz %l[error2]\n\t" +#endif + /* try store */ + LONG_S " %[newv2], %[v2]\n\t" + RSEQ_INJECT_ASM(5) + /* final store */ + LONG_S " %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, "", abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + /* try store input */ + [v2] "m" (*v2), + [newv2] "r" (newv2), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "memory", "cc", "r0" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +/* s390 is TSO. */ +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trymemcpy_storev)(intptr_t *v, in= tptr_t expect, + void *dst, void *src, size_t len, + intptr_t newv, int cpu) +{ + uint64_t rseq_scratch[3]; + + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) + RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) +#endif + LONG_S " %[src], %[rseq_scratch0]\n\t" + LONG_S " %[dst], %[rseq_scratch1]\n\t" + LONG_S " %[len], %[rseq_scratch2]\n\t" + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + LONG_CMP " %[expect], %[v]\n\t" + "jnz 5f\n\t" + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 6f) + LONG_CMP " %[expect], %[v]\n\t" + "jnz 7f\n\t" +#endif + /* try memcpy */ + LONG_LT_R " %[len], %[len]\n\t" + "jz 333f\n\t" + "222:\n\t" + "ic %%r0,0(%[src])\n\t" + "stc %%r0,0(%[dst])\n\t" + LONG_ADDI " %[src], 1\n\t" + LONG_ADDI " %[dst], 1\n\t" + LONG_ADDI " %[len], -1\n\t" + "jnz 222b\n\t" + "333:\n\t" + RSEQ_INJECT_ASM(5) + /* final store */ + LONG_S " %[newv], %[v]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(6) + /* teardown */ + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t" + RSEQ_ASM_DEFINE_ABORT(4, + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + abort) + RSEQ_ASM_DEFINE_CMPFAIL(5, + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + cmpfail) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_CMPFAIL(6, + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + error1) + RSEQ_ASM_DEFINE_CMPFAIL(7, + LONG_L " %[len], %[rseq_scratch2]\n\t" + LONG_L " %[dst], %[rseq_scratch1]\n\t" + LONG_L " %[src], %[rseq_scratch0]\n\t", + error2) +#endif + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (rseq_get_abi()->cpu_id), + [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), + /* final store input */ + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv), + /* try memcpy input */ + [dst] "r" (dst), + [src] "r" (src), + [len] "r" (len), + [rseq_scratch0] "m" (rseq_scratch[0]), + [rseq_scratch1] "m" (rseq_scratch[1]), + [rseq_scratch2] "m" (rseq_scratch[2]) + RSEQ_INJECT_INPUT + : "memory", "cc", "r0" + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +cmpfail: + rseq_after_asm_goto(); + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_after_asm_goto(); + rseq_bug("cpu_id comparison failed"); +error2: + rseq_after_asm_goto(); + rseq_bug("expected value comparison failed"); +#endif +} + +#endif /* #if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_= MO_RELEASE)) && + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) */ + +#include "rseq-bits-reset.h" diff --git a/tools/testing/selftests/rseq/rseq-s390.h b/tools/testing/selft= ests/rseq/rseq-s390.h index 4d3286453bbf..72c89a9b4098 100644 --- a/tools/testing/selftests/rseq/rseq-s390.h +++ b/tools/testing/selftests/rseq/rseq-s390.h @@ -130,476 +130,34 @@ do { \ "jg %l[" __rseq_str(cmpfail_label) "]\n\t" \ ".popsection\n\t" =20 -static inline __attribute__((always_inline)) -int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cp= u) -{ - RSEQ_INJECT_C(9) +/* Per-cpu-id indexing. */ =20 - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - LONG_CMP " %[expect], %[v]\n\t" - "jnz %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - LONG_CMP " %[expect], %[v]\n\t" - "jnz %l[error2]\n\t" -#endif - /* final store */ - LONG_S " %[newv], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(5) - RSEQ_ASM_DEFINE_ABORT(4, "", abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - RSEQ_INJECT_INPUT - : "memory", "cc", "r0" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -/* - * Compare @v against @expectnot. When it does _not_ match, load @v - * into @load, and store the content of *@v + voffp into @v. - */ -static inline __attribute__((always_inline)) -int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, - long voffp, intptr_t *load, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - LONG_L " %%r1, %[v]\n\t" - LONG_CMP_R " %%r1, %[expectnot]\n\t" - "je %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - LONG_L " %%r1, %[v]\n\t" - LONG_CMP_R " %%r1, %[expectnot]\n\t" - "je %l[error2]\n\t" -#endif - LONG_S " %%r1, %[load]\n\t" - LONG_ADD_R " %%r1, %[voffp]\n\t" - LONG_L " %%r1, 0(%%r1)\n\t" - /* final store */ - LONG_S " %%r1, %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(5) - RSEQ_ASM_DEFINE_ABORT(4, "", abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* final store input */ - [v] "m" (*v), - [expectnot] "r" (expectnot), - [voffp] "r" (voffp), - [load] "m" (*load) - RSEQ_INJECT_INPUT - : "memory", "cc", "r0", "r1" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_addv(intptr_t *v, intptr_t count, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) -#endif - LONG_L " %%r0, %[v]\n\t" - LONG_ADD_R " %%r0, %[count]\n\t" - /* final store */ - LONG_S " %%r0, %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(4) - RSEQ_ASM_DEFINE_ABORT(4, "", abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* final store input */ - [v] "m" (*v), - [count] "r" (count) - RSEQ_INJECT_INPUT - : "memory", "cc", "r0" - RSEQ_INJECT_CLOBBER - : abort -#ifdef RSEQ_COMPARE_TWICE - , error1 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -#endif -} +#define RSEQ_TEMPLATE_CPU_ID +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-s390-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED =20 -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t newv2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) +#define RSEQ_TEMPLATE_MO_RELEASE +#include "rseq-s390-bits.h" +#undef RSEQ_TEMPLATE_MO_RELEASE +#undef RSEQ_TEMPLATE_CPU_ID =20 - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - LONG_CMP " %[expect], %[v]\n\t" - "jnz %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - LONG_CMP " %[expect], %[v]\n\t" - "jnz %l[error2]\n\t" -#endif - /* try store */ - LONG_S " %[newv2], %[v2]\n\t" - RSEQ_INJECT_ASM(5) - /* final store */ - LONG_S " %[newv], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(6) - RSEQ_ASM_DEFINE_ABORT(4, "", abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* try store input */ - [v2] "m" (*v2), - [newv2] "r" (newv2), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - RSEQ_INJECT_INPUT - : "memory", "cc", "r0" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} +/* Per-vm-vcpu-id indexing. */ =20 -/* s390 is TSO. */ -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t newv2, - intptr_t newv, int cpu) -{ - return rseq_cmpeqv_trystorev_storev(v, expect, v2, newv2, newv, cpu); -} +#define RSEQ_TEMPLATE_VM_VCPU_ID +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-s390-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED =20 -static inline __attribute__((always_inline)) -int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t expect2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) +#define RSEQ_TEMPLATE_MO_RELEASE +#include "rseq-s390-bits.h" +#undef RSEQ_TEMPLATE_MO_RELEASE +#undef RSEQ_TEMPLATE_VM_VCPU_ID =20 - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error3]) -#endif - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - LONG_CMP " %[expect], %[v]\n\t" - "jnz %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(4) - LONG_CMP " %[expect2], %[v2]\n\t" - "jnz %l[cmpfail]\n\t" - RSEQ_INJECT_ASM(5) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1]) - LONG_CMP " %[expect], %[v]\n\t" - "jnz %l[error2]\n\t" - LONG_CMP " %[expect2], %[v2]\n\t" - "jnz %l[error3]\n\t" -#endif - /* final store */ - LONG_S " %[newv], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(6) - RSEQ_ASM_DEFINE_ABORT(4, "", abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* cmp2 input */ - [v2] "m" (*v2), - [expect2] "r" (expect2), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - RSEQ_INJECT_INPUT - : "memory", "cc", "r0" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2, error3 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("1st expected value comparison failed"); -error3: - rseq_after_asm_goto(); - rseq_bug("2nd expected value comparison failed"); -#endif -} - -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, - void *dst, void *src, size_t len, - intptr_t newv, int cpu) -{ - uint64_t rseq_scratch[3]; - - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto ( - RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[cmpfail]) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error1]) - RSEQ_ASM_DEFINE_EXIT_POINT(1f, %l[error2]) -#endif - LONG_S " %[src], %[rseq_scratch0]\n\t" - LONG_S " %[dst], %[rseq_scratch1]\n\t" - LONG_S " %[len], %[rseq_scratch2]\n\t" - /* Start rseq by storing table entry pointer into rseq_cs. */ - RSEQ_ASM_STORE_RSEQ_CS(1, 3b, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - LONG_CMP " %[expect], %[v]\n\t" - "jnz 5f\n\t" - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 6f) - LONG_CMP " %[expect], %[v]\n\t" - "jnz 7f\n\t" -#endif - /* try memcpy */ - LONG_LT_R " %[len], %[len]\n\t" - "jz 333f\n\t" - "222:\n\t" - "ic %%r0,0(%[src])\n\t" - "stc %%r0,0(%[dst])\n\t" - LONG_ADDI " %[src], 1\n\t" - LONG_ADDI " %[dst], 1\n\t" - LONG_ADDI " %[len], -1\n\t" - "jnz 222b\n\t" - "333:\n\t" - RSEQ_INJECT_ASM(5) - /* final store */ - LONG_S " %[newv], %[v]\n\t" - "2:\n\t" - RSEQ_INJECT_ASM(6) - /* teardown */ - LONG_L " %[len], %[rseq_scratch2]\n\t" - LONG_L " %[dst], %[rseq_scratch1]\n\t" - LONG_L " %[src], %[rseq_scratch0]\n\t" - RSEQ_ASM_DEFINE_ABORT(4, - LONG_L " %[len], %[rseq_scratch2]\n\t" - LONG_L " %[dst], %[rseq_scratch1]\n\t" - LONG_L " %[src], %[rseq_scratch0]\n\t", - abort) - RSEQ_ASM_DEFINE_CMPFAIL(5, - LONG_L " %[len], %[rseq_scratch2]\n\t" - LONG_L " %[dst], %[rseq_scratch1]\n\t" - LONG_L " %[src], %[rseq_scratch0]\n\t", - cmpfail) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_CMPFAIL(6, - LONG_L " %[len], %[rseq_scratch2]\n\t" - LONG_L " %[dst], %[rseq_scratch1]\n\t" - LONG_L " %[src], %[rseq_scratch0]\n\t", - error1) - RSEQ_ASM_DEFINE_CMPFAIL(7, - LONG_L " %[len], %[rseq_scratch2]\n\t" - LONG_L " %[dst], %[rseq_scratch1]\n\t" - LONG_L " %[src], %[rseq_scratch0]\n\t", - error2) -#endif - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (rseq_get_abi()->cpu_id), - [rseq_cs] "m" (rseq_get_abi()->rseq_cs.arch.ptr), - /* final store input */ - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv), - /* try memcpy input */ - [dst] "r" (dst), - [src] "r" (src), - [len] "r" (len), - [rseq_scratch0] "m" (rseq_scratch[0]), - [rseq_scratch1] "m" (rseq_scratch[1]), - [rseq_scratch2] "m" (rseq_scratch[2]) - RSEQ_INJECT_INPUT - : "memory", "cc", "r0" - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - rseq_after_asm_goto(); - return 0; -abort: - rseq_after_asm_goto(); - RSEQ_INJECT_FAILED - return -1; -cmpfail: - rseq_after_asm_goto(); - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_after_asm_goto(); - rseq_bug("cpu_id comparison failed"); -error2: - rseq_after_asm_goto(); - rseq_bug("expected value comparison failed"); -#endif -} +/* APIs which are not based on cpu ids. */ =20 -/* s390 is TSO. */ -static inline __attribute__((always_inline)) -int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, - void *dst, void *src, size_t len, - intptr_t newv, int cpu) -{ - return rseq_cmpeqv_trymemcpy_storev(v, expect, dst, src, len, - newv, cpu); -} +#define RSEQ_TEMPLATE_CPU_ID_NONE +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-s390-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED +#undef RSEQ_TEMPLATE_CPU_ID_NONE --=20 2.17.1 From nobody Tue Apr 14 22:46:25 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46867C00144 for ; Fri, 29 Jul 2022 19:04:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238995AbiG2TEX (ORCPT ); Fri, 29 Jul 2022 15:04:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52808 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238789AbiG2TDo (ORCPT ); Fri, 29 Jul 2022 15:03:44 -0400 Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F20AE8AB19; Fri, 29 Jul 2022 12:03:02 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id D6FB9318EA5; Fri, 29 Jul 2022 15:03:00 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id aHErRqhBMWOb; Fri, 29 Jul 2022 15:02:59 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 0BFB6319091; Fri, 29 Jul 2022 15:02:52 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 0BFB6319091 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1659121372; bh=jzkCnmhhc3qVfUYlcY04+wWPx92BK3WW+08Dpv5MDng=; h=From:To:Date:Message-Id; b=ls3rI8Cu8wUVnWFdDUxerkMsCkQOihvBqOi+pXA39VscQ7ktCrI+VPaSzgvvhZHBc mEt1Ncsd7vVNqqsO5eFXmGnndm7/7wJeixr/5LQ+CCfJSWQsMeN3Sgr1JXt5vFkZfX kwxnFSCNcftqBWHMqs6j0x33rFDbsl1JrOX4xdu/7yQZ8Jqr8MqxL3hVO4CLrbgdSn byDlkbyNmRuItnHwynHi8dBBdsYm39HoW+UXt6lrxkiuM8fuon1g3nJTcnJW1lWsJr P72Bm+5SadtU31uvfC75/vL2nzDcUUUBzJyo/d6g0HNB1opBu7LIudBL0DzHQySQpi qutycQDf14gSw== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id deUGm8uTrW2Z; Fri, 29 Jul 2022 15:02:51 -0400 (EDT) Received: from localhost.localdomain (192-222-180-24.qc.cable.ebox.net [192.222.180.24]) by mail.efficios.com (Postfix) with ESMTPSA id 7ADB6318B56; Fri, 29 Jul 2022 15:02:47 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Thomas Gleixner , "Paul E . McKenney" , Boqun Feng , "H . Peter Anvin" , Paul Turner , linux-api@vger.kernel.org, Christian Brauner , Florian Weimer , David.Laight@ACULAB.COM, carlos@redhat.com, Peter Oskolkov , Mathieu Desnoyers , Vincent Chen , Eric Lin , Palmer Dabbelt Subject: [PATCH v3 19/23] selftests/rseq: riscv: Template memory ordering and percpu access mode Date: Fri, 29 Jul 2022 15:02:21 -0400 Message-Id: <20220729190225.12726-20-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> References: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Introduce a rseq-riscv-bits.h template header which is internally included to generate the static inline functions covering: - relaxed and release memory ordering, - per-cpu-id and per-vm-vcpu-id per-cpu data access. Signed-off-by: Mathieu Desnoyers Cc: Vincent Chen Cc: Eric Lin Cc: Palmer Dabbelt --- .../testing/selftests/rseq/rseq-riscv-bits.h | 410 ++++++++++++++ tools/testing/selftests/rseq/rseq-riscv.h | 527 +----------------- 2 files changed, 437 insertions(+), 500 deletions(-) create mode 100644 tools/testing/selftests/rseq/rseq-riscv-bits.h diff --git a/tools/testing/selftests/rseq/rseq-riscv-bits.h b/tools/testing= /selftests/rseq/rseq-riscv-bits.h new file mode 100644 index 000000000000..e42211afdef0 --- /dev/null +++ b/tools/testing/selftests/rseq/rseq-riscv-bits.h @@ -0,0 +1,410 @@ +/* SPDX-License-Identifier: LGPL-2.1 OR MIT */ + +#include "rseq-bits-template.h" + +#if defined(RSEQ_TEMPLATE_MO_RELAXED) && \ + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) + +static inline __always_inline +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_storev)(intptr_t *v, intptr_t exp= ect, intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]") +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]") + RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]") +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPEQ(v, expect, "%l[cmpfail]") + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]") + RSEQ_ASM_OP_CMPEQ(v, expect, "%l[error2]") +#endif + RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) + RSEQ_INJECT_ASM(5) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [v] "m" (*v), + [expect] "r" (expect), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG_1 + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __always_inline +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpnev_storeoffp_load)(intptr_t *v, intp= tr_t expectnot, + off_t voffp, intptr_t *load, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]") +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]") + RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]") +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPNE(v, expectnot, "%l[cmpfail]") + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]") + RSEQ_ASM_OP_CMPNE(v, expectnot, "%l[error2]") +#endif + RSEQ_ASM_OP_R_LOAD(v) + RSEQ_ASM_OP_R_STORE(load) + RSEQ_ASM_OP_R_LOAD_OFF(voffp) + RSEQ_ASM_OP_R_FINAL_STORE(v, 3) + RSEQ_INJECT_ASM(5) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [v] "m" (*v), + [expectnot] "r" (expectnot), + [load] "m" (*load), + [voffp] "r" (voffp) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG_1 + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __always_inline +int RSEQ_TEMPLATE_IDENTIFIER(rseq_addv)(intptr_t *v, intptr_t count, int c= pu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]") +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]") +#endif + RSEQ_ASM_OP_R_LOAD(v) + RSEQ_ASM_OP_R_ADD(count) + RSEQ_ASM_OP_R_FINAL_STORE(v, 3) + RSEQ_INJECT_ASM(4) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [v] "m" (*v), + [count] "r" (count) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG_1 + RSEQ_INJECT_CLOBBER + : abort +#ifdef RSEQ_COMPARE_TWICE + , error1 +#endif + ); + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +#endif +} + +static inline __always_inline +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_cmpeqv_storev)(intptr_t *v, intpt= r_t expect, + intptr_t *v2, intptr_t expect2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]") +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]") + RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]") + RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error3]") +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPEQ(v, expect, "%l[cmpfail]") + RSEQ_INJECT_ASM(4) + RSEQ_ASM_OP_CMPEQ(v2, expect2, "%l[cmpfail]") + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]") + RSEQ_ASM_OP_CMPEQ(v, expect, "%l[error2]") + RSEQ_ASM_OP_CMPEQ(v2, expect2, "%l[error3]") +#endif + RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [v] "m" (*v), + [expect] "r" (expect), + [v2] "m" (*v2), + [expect2] "r" (expect2), + [newv] "r" (newv) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG_1 + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2, error3 +#endif + ); + + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +error3: + rseq_bug("2nd expected value comparison failed"); +#endif +} + +#define RSEQ_ARCH_HAS_OFFSET_DEREF_ADDV + +/* + * pval =3D *(ptr+off) + * *pval +=3D inc; + */ +static inline __always_inline +int RSEQ_TEMPLATE_IDENTIFIER(rseq_offset_deref_addv)(intptr_t *ptr, off_t = off, intptr_t inc, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]") +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]") +#endif + RSEQ_ASM_OP_R_DEREF_ADDV(ptr, off, 3) + RSEQ_INJECT_ASM(4) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [ptr] "r" (ptr), + [off] "er" (off), + [inc] "er" (inc) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG_1 + RSEQ_INJECT_CLOBBER + : abort +#ifdef RSEQ_COMPARE_TWICE + , error1 +#endif + ); + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +#endif +} + +#endif /* #if defined(RSEQ_TEMPLATE_MO_RELAXED) && + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) */ + +#if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_MO_RELEASE= )) && \ + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) + +static inline __always_inline +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trystorev_storev)(intptr_t *v, in= tptr_t expect, + intptr_t *v2, intptr_t newv2, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]") +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]") + RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]") +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPEQ(v, expect, "%l[cmpfail]") + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]") + RSEQ_ASM_OP_CMPEQ(v, expect, "%l[error2]") +#endif + RSEQ_ASM_OP_STORE(newv2, v2) + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_TEMPLATE_MO_RELEASE + RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3) +#else + RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) +#endif + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [expect] "r" (expect), + [v] "m" (*v), + [newv] "r" (newv), + [v2] "m" (*v2), + [newv2] "r" (newv2) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG_1 + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +static inline __always_inline +int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trymemcpy_storev)(intptr_t *v, in= tptr_t expect, + void *dst, void *src, size_t len, + intptr_t newv, int cpu) +{ + RSEQ_INJECT_C(9) + __asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) + RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]") +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]") + RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]") +#endif + RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) + RSEQ_INJECT_ASM(3) + RSEQ_ASM_OP_CMPEQ(v, expect, "%l[cmpfail]") + RSEQ_INJECT_ASM(4) +#ifdef RSEQ_COMPARE_TWICE + RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]") + RSEQ_ASM_OP_CMPEQ(v, expect, "%l[error2]") +#endif + RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len) + RSEQ_INJECT_ASM(5) +#ifdef RSEQ_TEMPLATE_MO_RELEASE + RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3) +#else + RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) +#endif + RSEQ_INJECT_ASM(6) + RSEQ_ASM_DEFINE_ABORT(4, abort) + : /* gcc asm goto does not allow outputs */ + : [cpu_id] "r" (cpu), + [current_cpu_id] "m" (__rseq_abi.cpu_id), + [rseq_cs] "m" (__rseq_abi.rseq_cs), + [expect] "r" (expect), + [v] "m" (*v), + [newv] "r" (newv), + [dst] "r" (dst), + [src] "r" (src), + [len] "r" (len) + RSEQ_INJECT_INPUT + : "memory", RSEQ_ASM_TMP_REG_1, RSEQ_ASM_TMP_REG_2, + RSEQ_ASM_TMP_REG_3, RSEQ_ASM_TMP_REG_4 + RSEQ_INJECT_CLOBBER + : abort, cmpfail +#ifdef RSEQ_COMPARE_TWICE + , error1, error2 +#endif + ); + + return 0; +abort: + RSEQ_INJECT_FAILED + return -1; +cmpfail: + return 1; +#ifdef RSEQ_COMPARE_TWICE +error1: + rseq_bug("cpu_id comparison failed"); +error2: + rseq_bug("expected value comparison failed"); +#endif +} + +#endif /* #if (defined(RSEQ_TEMPLATE_MO_RELAXED) || defined(RSEQ_TEMPLATE_= MO_RELEASE)) && + (defined(RSEQ_TEMPLATE_CPU_ID) || defined(RSEQ_TEMPLATE_VM_VCPU_ID)) */ + +#include "rseq-bits-reset.h" diff --git a/tools/testing/selftests/rseq/rseq-riscv.h b/tools/testing/self= tests/rseq/rseq-riscv.h index 3394b8f7f322..57eaa491b260 100644 --- a/tools/testing/selftests/rseq/rseq-riscv.h +++ b/tools/testing/selftests/rseq/rseq-riscv.h @@ -165,507 +165,34 @@ do { \ RSEQ_ASM_OP_R_ADD(inc) \ __rseq_str(post_commit_label) ":\n" =20 -static inline __always_inline -int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cp= u) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]") -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]") - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]") -#endif - RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - RSEQ_ASM_OP_CMPEQ(v, expect, "%l[cmpfail]") - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]") - RSEQ_ASM_OP_CMPEQ(v, expect, "%l[error2]") -#endif - RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) - RSEQ_INJECT_ASM(5) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), - [v] "m" (*v), - [expect] "r" (expect), - [newv] "r" (newv) - RSEQ_INJECT_INPUT - : "memory", RSEQ_ASM_TMP_REG_1 - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - - return 0; -abort: - RSEQ_INJECT_FAILED - return -1; -cmpfail: - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_bug("cpu_id comparison failed"); -error2: - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __always_inline -int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot, - off_t voffp, intptr_t *load, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]") -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]") - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]") -#endif - RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - RSEQ_ASM_OP_CMPNE(v, expectnot, "%l[cmpfail]") - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]") - RSEQ_ASM_OP_CMPNE(v, expectnot, "%l[error2]") -#endif - RSEQ_ASM_OP_R_LOAD(v) - RSEQ_ASM_OP_R_STORE(load) - RSEQ_ASM_OP_R_LOAD_OFF(voffp) - RSEQ_ASM_OP_R_FINAL_STORE(v, 3) - RSEQ_INJECT_ASM(5) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), - [v] "m" (*v), - [expectnot] "r" (expectnot), - [load] "m" (*load), - [voffp] "r" (voffp) - RSEQ_INJECT_INPUT - : "memory", RSEQ_ASM_TMP_REG_1 - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - return 0; -abort: - RSEQ_INJECT_FAILED - return -1; -cmpfail: - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_bug("cpu_id comparison failed"); -error2: - rseq_bug("expected value comparison failed"); -#endif -} +/* Per-cpu-id indexing. */ =20 -static inline __always_inline -int rseq_addv(intptr_t *v, intptr_t count, int cpu) -{ - RSEQ_INJECT_C(9) +#define RSEQ_TEMPLATE_CPU_ID +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-riscv-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED =20 - __asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]") -#endif - RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]") -#endif - RSEQ_ASM_OP_R_LOAD(v) - RSEQ_ASM_OP_R_ADD(count) - RSEQ_ASM_OP_R_FINAL_STORE(v, 3) - RSEQ_INJECT_ASM(4) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), - [v] "m" (*v), - [count] "r" (count) - RSEQ_INJECT_INPUT - : "memory", RSEQ_ASM_TMP_REG_1 - RSEQ_INJECT_CLOBBER - : abort -#ifdef RSEQ_COMPARE_TWICE - , error1 -#endif - ); - return 0; -abort: - RSEQ_INJECT_FAILED - return -1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_bug("cpu_id comparison failed"); -#endif -} - -static inline __always_inline -int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t newv2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]") -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]") - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]") -#endif - RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - RSEQ_ASM_OP_CMPEQ(v, expect, "%l[cmpfail]") - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]") - RSEQ_ASM_OP_CMPEQ(v, expect, "%l[error2]") -#endif - RSEQ_ASM_OP_STORE(newv2, v2) - RSEQ_INJECT_ASM(5) - RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) - RSEQ_INJECT_ASM(6) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), - [expect] "r" (expect), - [v] "m" (*v), - [newv] "r" (newv), - [v2] "m" (*v2), - [newv2] "r" (newv2) - RSEQ_INJECT_INPUT - : "memory", RSEQ_ASM_TMP_REG_1 - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - - return 0; -abort: - RSEQ_INJECT_FAILED - return -1; -cmpfail: - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_bug("cpu_id comparison failed"); -error2: - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __always_inline -int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t newv2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]") -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]") - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]") -#endif - RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - RSEQ_ASM_OP_CMPEQ(v, expect, "%l[cmpfail]") - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]") - RSEQ_ASM_OP_CMPEQ(v, expect, "%l[error2]") -#endif - RSEQ_ASM_OP_STORE(newv2, v2) - RSEQ_INJECT_ASM(5) - RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3) - RSEQ_INJECT_ASM(6) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), - [expect] "r" (expect), - [v] "m" (*v), - [newv] "r" (newv), - [v2] "m" (*v2), - [newv2] "r" (newv2) - RSEQ_INJECT_INPUT - : "memory", RSEQ_ASM_TMP_REG_1 - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - - return 0; -abort: - RSEQ_INJECT_FAILED - return -1; -cmpfail: - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_bug("cpu_id comparison failed"); -error2: - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __always_inline -int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect, - intptr_t *v2, intptr_t expect2, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]") -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]") - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]") - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error3]") -#endif - RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - RSEQ_ASM_OP_CMPEQ(v, expect, "%l[cmpfail]") - RSEQ_INJECT_ASM(4) - RSEQ_ASM_OP_CMPEQ(v2, expect2, "%l[cmpfail]") - RSEQ_INJECT_ASM(5) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]") - RSEQ_ASM_OP_CMPEQ(v, expect, "%l[error2]") - RSEQ_ASM_OP_CMPEQ(v2, expect2, "%l[error3]") -#endif - RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) - RSEQ_INJECT_ASM(6) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), - [v] "m" (*v), - [expect] "r" (expect), - [v2] "m" (*v2), - [expect2] "r" (expect2), - [newv] "r" (newv) - RSEQ_INJECT_INPUT - : "memory", RSEQ_ASM_TMP_REG_1 - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2, error3 -#endif - ); - - return 0; -abort: - RSEQ_INJECT_FAILED - return -1; -cmpfail: - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_bug("cpu_id comparison failed"); -error2: - rseq_bug("expected value comparison failed"); -error3: - rseq_bug("2nd expected value comparison failed"); -#endif -} - -static inline __always_inline -int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect, - void *dst, void *src, size_t len, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - __asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]") -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]") - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]") -#endif - RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - RSEQ_ASM_OP_CMPEQ(v, expect, "%l[cmpfail]") - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]") - RSEQ_ASM_OP_CMPEQ(v, expect, "%l[error2]") -#endif - RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len) - RSEQ_INJECT_ASM(5) - RSEQ_ASM_OP_FINAL_STORE(newv, v, 3) - RSEQ_INJECT_ASM(6) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), - [expect] "r" (expect), - [v] "m" (*v), - [newv] "r" (newv), - [dst] "r" (dst), - [src] "r" (src), - [len] "r" (len) - RSEQ_INJECT_INPUT - : "memory", RSEQ_ASM_TMP_REG_1, RSEQ_ASM_TMP_REG_2, - RSEQ_ASM_TMP_REG_3, RSEQ_ASM_TMP_REG_4 - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - - return 0; -abort: - RSEQ_INJECT_FAILED - return -1; -cmpfail: - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_bug("cpu_id comparison failed"); -error2: - rseq_bug("expected value comparison failed"); -#endif -} - -static inline __always_inline -int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect, - void *dst, void *src, size_t len, - intptr_t newv, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[cmpfail]") -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]") - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error2]") -#endif - RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) - RSEQ_ASM_OP_CMPEQ(v, expect, "%l[cmpfail]") - RSEQ_INJECT_ASM(4) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]") - RSEQ_ASM_OP_CMPEQ(v, expect, "%l[error2]") -#endif - RSEQ_ASM_OP_R_BAD_MEMCPY(dst, src, len) - RSEQ_INJECT_ASM(5) - RSEQ_ASM_OP_FINAL_STORE_RELEASE(newv, v, 3) - RSEQ_INJECT_ASM(6) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), - [expect] "r" (expect), - [v] "m" (*v), - [newv] "r" (newv), - [dst] "r" (dst), - [src] "r" (src), - [len] "r" (len) - RSEQ_INJECT_INPUT - : "memory", RSEQ_ASM_TMP_REG_1, RSEQ_ASM_TMP_REG_2, - RSEQ_ASM_TMP_REG_3, RSEQ_ASM_TMP_REG_4 - RSEQ_INJECT_CLOBBER - : abort, cmpfail -#ifdef RSEQ_COMPARE_TWICE - , error1, error2 -#endif - ); - - return 0; -abort: - RSEQ_INJECT_FAILED - return -1; -cmpfail: - return 1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_bug("cpu_id comparison failed"); -error2: - rseq_bug("expected value comparison failed"); -#endif -} +#define RSEQ_TEMPLATE_MO_RELEASE +#include "rseq-riscv-bits.h" +#undef RSEQ_TEMPLATE_MO_RELEASE +#undef RSEQ_TEMPLATE_CPU_ID =20 -#define RSEQ_ARCH_HAS_OFFSET_DEREF_ADDV +/* Per-vm-vcpu-id indexing. */ =20 -/* - * pval =3D *(ptr+off) - * *pval +=3D inc; - */ -static inline __always_inline -int rseq_offset_deref_addv(intptr_t *ptr, off_t off, intptr_t inc, int cpu) -{ - RSEQ_INJECT_C(9) - - __asm__ __volatile__ goto(RSEQ_ASM_DEFINE_TABLE(1, 2f, 3f, 4f) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_DEFINE_EXIT_POINT(2f, "%l[error1]") -#endif - RSEQ_ASM_STORE_RSEQ_CS(2, 1b, rseq_cs) - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f) - RSEQ_INJECT_ASM(3) -#ifdef RSEQ_COMPARE_TWICE - RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, "%l[error1]") -#endif - RSEQ_ASM_OP_R_DEREF_ADDV(ptr, off, 3) - RSEQ_INJECT_ASM(4) - RSEQ_ASM_DEFINE_ABORT(4, abort) - : /* gcc asm goto does not allow outputs */ - : [cpu_id] "r" (cpu), - [current_cpu_id] "m" (__rseq_abi.cpu_id), - [rseq_cs] "m" (__rseq_abi.rseq_cs), - [ptr] "r" (ptr), - [off] "er" (off), - [inc] "er" (inc) - RSEQ_INJECT_INPUT - : "memory", RSEQ_ASM_TMP_REG_1 - RSEQ_INJECT_CLOBBER - : abort -#ifdef RSEQ_COMPARE_TWICE - , error1 -#endif - ); - return 0; -abort: - RSEQ_INJECT_FAILED - return -1; -#ifdef RSEQ_COMPARE_TWICE -error1: - rseq_bug("cpu_id comparison failed"); -#endif -} +#define RSEQ_TEMPLATE_VM_VCPU_ID +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-riscv-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED + +#define RSEQ_TEMPLATE_MO_RELEASE +#include "rseq-riscv-bits.h" +#undef RSEQ_TEMPLATE_MO_RELEASE +#undef RSEQ_TEMPLATE_VM_VCPU_ID + +/* APIs which are not based on cpu ids. */ + +#define RSEQ_TEMPLATE_CPU_ID_NONE +#define RSEQ_TEMPLATE_MO_RELAXED +#include "rseq-riscv-bits.h" +#undef RSEQ_TEMPLATE_MO_RELAXED +#undef RSEQ_TEMPLATE_CPU_ID_NONE --=20 2.17.1 From nobody Tue Apr 14 22:46:25 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F8E0C00144 for ; Fri, 29 Jul 2022 19:04:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238975AbiG2TEL (ORCPT ); Fri, 29 Jul 2022 15:04:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52806 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238650AbiG2TDo (ORCPT ); Fri, 29 Jul 2022 15:03:44 -0400 Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D209D88E29; Fri, 29 Jul 2022 12:03:02 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 2B08E3190A2; Fri, 29 Jul 2022 15:03:01 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id OtB2fgSjqc2M; Fri, 29 Jul 2022 15:03:00 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 3A39D319023; Fri, 29 Jul 2022 15:02:52 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 3A39D319023 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1659121372; bh=GTblBPB6Z8Hr33bZ2XwCDwRC9BmLvYt9Eup9e+epf08=; h=From:To:Date:Message-Id; b=QNoob/j0qlJ2m63Ypw+GhGWfvL5vVuewdAaYFqRBIkrCeJjnY32B+yNdvzC5IxNZm 50LXYb6SnsUXD0r76dwwrmd+qElKUcWBOqXB+tHA7QggH6p8M/GDzyuWSRwaFNbKWg xWFzrkdCtUW7f3dgapvrSMgbqJ9TbEb8IQTyH30Ir7TLDAQD/tgmr1XYLm4JnHEM/C yipQjzEHU6hAfy6P5ye9PejYMV5RsqD8SE89RdUJ9Q/+sGk/A//Hb9l0dtxo2/M5Qr NMUS5xI+xE34yo+SSOI6VACo9n+vzRa9N+I9rdVtE42QbTYmLY+B36eZnC4R7QNMEy 4vqzqgTnSul2A== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 0MUj4-wnFCBI; Fri, 29 Jul 2022 15:02:52 -0400 (EDT) Received: from localhost.localdomain (192-222-180-24.qc.cable.ebox.net [192.222.180.24]) by mail.efficios.com (Postfix) with ESMTPSA id DFDB331908E; Fri, 29 Jul 2022 15:02:47 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Thomas Gleixner , "Paul E . McKenney" , Boqun Feng , "H . Peter Anvin" , Paul Turner , linux-api@vger.kernel.org, Christian Brauner , Florian Weimer , David.Laight@ACULAB.COM, carlos@redhat.com, Peter Oskolkov , Mathieu Desnoyers Subject: [PATCH v3 20/23] selftests/rseq: basic percpu ops vm_vcpu_id test Date: Fri, 29 Jul 2022 15:02:22 -0400 Message-Id: <20220729190225.12726-21-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> References: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Adapt to the rseq.h API changes introduced by commits "selftests/rseq: : Template memory ordering and percpu access mode". Build a new basic_percpu_ops_vm_vcpu_id_test to test the new "vm_vcpu_id" rseq field. Signed-off-by: Mathieu Desnoyers --- tools/testing/selftests/rseq/.gitignore | 1 + tools/testing/selftests/rseq/Makefile | 5 +- .../selftests/rseq/basic_percpu_ops_test.c | 46 ++++++++++++++++--- 3 files changed, 44 insertions(+), 8 deletions(-) diff --git a/tools/testing/selftests/rseq/.gitignore b/tools/testing/selfte= sts/rseq/.gitignore index 5910888ebfe1..5a7e5acc628c 100644 --- a/tools/testing/selftests/rseq/.gitignore +++ b/tools/testing/selftests/rseq/.gitignore @@ -1,5 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only basic_percpu_ops_test +basic_percpu_ops_vm_vcpu_id_test basic_test basic_rseq_op_test param_test diff --git a/tools/testing/selftests/rseq/Makefile b/tools/testing/selftest= s/rseq/Makefile index 215e1067f037..4210c135e621 100644 --- a/tools/testing/selftests/rseq/Makefile +++ b/tools/testing/selftests/rseq/Makefile @@ -12,7 +12,7 @@ LDLIBS +=3D -lpthread -ldl # still track changes to header files and depend on shared object. OVERRIDE_TARGETS =3D 1 =20 -TEST_GEN_PROGS =3D basic_test basic_percpu_ops_test param_test \ +TEST_GEN_PROGS =3D basic_test basic_percpu_ops_test basic_percpu_ops_vm_vc= pu_id_test param_test \ param_test_benchmark param_test_compare_twice =20 TEST_GEN_PROGS_EXTENDED =3D librseq.so @@ -29,6 +29,9 @@ $(OUTPUT)/librseq.so: rseq.c rseq.h rseq-*.h $(OUTPUT)/%: %.c $(TEST_GEN_PROGS_EXTENDED) rseq.h rseq-*.h $(CC) $(CFLAGS) $< $(LDLIBS) -lrseq -o $@ =20 +$(OUTPUT)/basic_percpu_ops_vm_vcpu_id_test: basic_percpu_ops_test.c $(TEST= _GEN_PROGS_EXTENDED) rseq.h rseq-*.h + $(CC) $(CFLAGS) -DBUILDOPT_RSEQ_PERCPU_VM_VCPU_ID $< $(LDLIBS) -lrseq -o = $@ + $(OUTPUT)/param_test_benchmark: param_test.c $(TEST_GEN_PROGS_EXTENDED) \ rseq.h rseq-*.h $(CC) $(CFLAGS) -DBENCHMARK $< $(LDLIBS) -lrseq -o $@ diff --git a/tools/testing/selftests/rseq/basic_percpu_ops_test.c b/tools/t= esting/selftests/rseq/basic_percpu_ops_test.c index 517756afc2a4..719ff9910e23 100644 --- a/tools/testing/selftests/rseq/basic_percpu_ops_test.c +++ b/tools/testing/selftests/rseq/basic_percpu_ops_test.c @@ -12,6 +12,32 @@ #include "../kselftest.h" #include "rseq.h" =20 +#ifdef BUILDOPT_RSEQ_PERCPU_VM_VCPU_ID +# define RSEQ_PERCPU RSEQ_PERCPU_VM_VCPU_ID +static +int get_current_cpu_id(void) +{ + return rseq_current_vm_vcpu_id(); +} +static +bool rseq_validate_cpu_id(void) +{ + return rseq_vm_vcpu_id_available(); +} +#else +# define RSEQ_PERCPU RSEQ_PERCPU_CPU_ID +static +int get_current_cpu_id(void) +{ + return rseq_cpu_start(); +} +static +bool rseq_validate_cpu_id(void) +{ + return rseq_current_cpu_raw() >=3D 0; +} +#endif + struct percpu_lock_entry { intptr_t v; } __attribute__((aligned(128))); @@ -51,9 +77,9 @@ int rseq_this_cpu_lock(struct percpu_lock *lock) for (;;) { int ret; =20 - cpu =3D rseq_cpu_start(); - ret =3D rseq_cmpeqv_storev(&lock->c[cpu].v, - 0, 1, cpu); + cpu =3D get_current_cpu_id(); + ret =3D rseq_cmpeqv_storev(RSEQ_MO_RELAXED, RSEQ_PERCPU, + &lock->c[cpu].v, 0, 1, cpu); if (rseq_likely(!ret)) break; /* Retry if comparison fails or rseq aborts. */ @@ -141,13 +167,14 @@ void this_cpu_list_push(struct percpu_list *list, intptr_t *targetptr, newval, expect; int ret; =20 - cpu =3D rseq_cpu_start(); + cpu =3D get_current_cpu_id(); /* Load list->c[cpu].head with single-copy atomicity. */ expect =3D (intptr_t)RSEQ_READ_ONCE(list->c[cpu].head); newval =3D (intptr_t)node; targetptr =3D (intptr_t *)&list->c[cpu].head; node->next =3D (struct percpu_list_node *)expect; - ret =3D rseq_cmpeqv_storev(targetptr, expect, newval, cpu); + ret =3D rseq_cmpeqv_storev(RSEQ_MO_RELAXED, RSEQ_PERCPU, + targetptr, expect, newval, cpu); if (rseq_likely(!ret)) break; /* Retry if comparison fails or rseq aborts. */ @@ -170,12 +197,13 @@ struct percpu_list_node *this_cpu_list_pop(struct per= cpu_list *list, long offset; int ret, cpu; =20 - cpu =3D rseq_cpu_start(); + cpu =3D get_current_cpu_id(); targetptr =3D (intptr_t *)&list->c[cpu].head; expectnot =3D (intptr_t)NULL; offset =3D offsetof(struct percpu_list_node, next); load =3D (intptr_t *)&head; - ret =3D rseq_cmpnev_storeoffp_load(targetptr, expectnot, + ret =3D rseq_cmpnev_storeoffp_load(RSEQ_MO_RELAXED, RSEQ_PERCPU, + targetptr, expectnot, offset, load, cpu); if (rseq_likely(!ret)) { if (_cpu) @@ -295,6 +323,10 @@ int main(int argc, char **argv) errno, strerror(errno)); goto error; } + if (!rseq_validate_cpu_id()) { + fprintf(stderr, "Error: cpu id getter unavailable\n"); + goto error; + } printf("spinlock\n"); test_percpu_spinlock(); printf("percpu_list\n"); --=20 2.17.1 From nobody Tue Apr 14 22:46:25 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B5F3C04A68 for ; Fri, 29 Jul 2022 19:04:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238772AbiG2TEa (ORCPT ); Fri, 29 Jul 2022 15:04:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51968 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238822AbiG2TDp (ORCPT ); Fri, 29 Jul 2022 15:03:45 -0400 Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0D0D58AB32; Fri, 29 Jul 2022 12:03:04 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id E0435319191; Fri, 29 Jul 2022 15:03:01 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id DorXpzbBMxQj; Fri, 29 Jul 2022 15:03:01 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 6D123319092; Fri, 29 Jul 2022 15:02:52 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 6D123319092 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1659121372; bh=XkmpOarcdJW1UW2Vu6h+4fz3fAgi3/rWuEJNTRHvayQ=; h=From:To:Date:Message-Id; b=QhF102CjUXNx3m/H3L8r7iEc0HrbQqGbQ+JCw8fXskzUp6EDvaSPTESpRtxsVg4Ep rpUSqGlU4KqmZGYcHFf1SXmEytndIiUxch5kMuHM4KPERGWpDy333XKe3GxUF/DwZ6 ZdkMOMgxV7PubMcFIW5fKUXD/qudDgzL49QQpfWd+VF/F+rbTZ7VLDvKJ0kjFT9VYN fTG1FwUgOwEpU32eo5S20lGUCEQ1NCVmEVFCujTorjVmTSBk9QLN7agjO7XJmgDA/C GsZrgIplhABY1ykTbPBAD1+lftT1bYZagnxMEMBJr8XGi2pE2yZNScoGJf1KHpvuSX 5LabHpQ2lvxbQ== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id m5TRdSfxwDs8; Fri, 29 Jul 2022 15:02:52 -0400 (EDT) Received: from localhost.localdomain (192-222-180-24.qc.cable.ebox.net [192.222.180.24]) by mail.efficios.com (Postfix) with ESMTPSA id 4058A318A7D; Fri, 29 Jul 2022 15:02:48 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Thomas Gleixner , "Paul E . McKenney" , Boqun Feng , "H . Peter Anvin" , Paul Turner , linux-api@vger.kernel.org, Christian Brauner , Florian Weimer , David.Laight@ACULAB.COM, carlos@redhat.com, Peter Oskolkov , Mathieu Desnoyers Subject: [PATCH v3 21/23] selftests/rseq: parametrized vm_vcpu_id test Date: Fri, 29 Jul 2022 15:02:23 -0400 Message-Id: <20220729190225.12726-22-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> References: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Adapt to the rseq.h API changes introduced by commits "selftests/rseq: : Template memory ordering and percpu access mode". Build a new param_test_vm_vcpu_id, param_test_vm_vcpu_id_benchmark, and param_test_vm_vcpu_id_compare_twice executables to test the new "vm_vcpu_id" rseq field. Signed-off-by: Mathieu Desnoyers --- tools/testing/selftests/rseq/.gitignore | 3 + tools/testing/selftests/rseq/Makefile | 15 +- tools/testing/selftests/rseq/param_test.c | 148 ++++++++++++------ .../testing/selftests/rseq/run_param_test.sh | 5 + 4 files changed, 122 insertions(+), 49 deletions(-) diff --git a/tools/testing/selftests/rseq/.gitignore b/tools/testing/selfte= sts/rseq/.gitignore index 5a7e5acc628c..db5c1a124c6c 100644 --- a/tools/testing/selftests/rseq/.gitignore +++ b/tools/testing/selftests/rseq/.gitignore @@ -6,3 +6,6 @@ basic_rseq_op_test param_test param_test_benchmark param_test_compare_twice +param_test_vm_vcpu_id +param_test_vm_vcpu_id_benchmark +param_test_vm_vcpu_id_compare_twice diff --git a/tools/testing/selftests/rseq/Makefile b/tools/testing/selftest= s/rseq/Makefile index 4210c135e621..3eec8e166385 100644 --- a/tools/testing/selftests/rseq/Makefile +++ b/tools/testing/selftests/rseq/Makefile @@ -13,7 +13,8 @@ LDLIBS +=3D -lpthread -ldl OVERRIDE_TARGETS =3D 1 =20 TEST_GEN_PROGS =3D basic_test basic_percpu_ops_test basic_percpu_ops_vm_vc= pu_id_test param_test \ - param_test_benchmark param_test_compare_twice + param_test_benchmark param_test_compare_twice param_test_vm_vcpu_id \ + param_test_vm_vcpu_id_benchmark param_test_vm_vcpu_id_compare_twice =20 TEST_GEN_PROGS_EXTENDED =3D librseq.so =20 @@ -39,3 +40,15 @@ $(OUTPUT)/param_test_benchmark: param_test.c $(TEST_GEN_= PROGS_EXTENDED) \ $(OUTPUT)/param_test_compare_twice: param_test.c $(TEST_GEN_PROGS_EXTENDED= ) \ rseq.h rseq-*.h $(CC) $(CFLAGS) -DRSEQ_COMPARE_TWICE $< $(LDLIBS) -lrseq -o $@ + +$(OUTPUT)/param_test_vm_vcpu_id: param_test.c $(TEST_GEN_PROGS_EXTENDED) \ + rseq.h rseq-*.h + $(CC) $(CFLAGS) -DBUILDOPT_RSEQ_PERCPU_VM_VCPU_ID $< $(LDLIBS) -lrseq -o = $@ + +$(OUTPUT)/param_test_vm_vcpu_id_benchmark: param_test.c $(TEST_GEN_PROGS_E= XTENDED) \ + rseq.h rseq-*.h + $(CC) $(CFLAGS) -DBUILDOPT_RSEQ_PERCPU_VM_VCPU_ID -DBENCHMARK $< $(LDLIBS= ) -lrseq -o $@ + +$(OUTPUT)/param_test_vm_vcpu_id_compare_twice: param_test.c $(TEST_GEN_PRO= GS_EXTENDED) \ + rseq.h rseq-*.h + $(CC) $(CFLAGS) -DBUILDOPT_RSEQ_PERCPU_VM_VCPU_ID -DRSEQ_COMPARE_TWICE $<= $(LDLIBS) -lrseq -o $@ diff --git a/tools/testing/selftests/rseq/param_test.c b/tools/testing/self= tests/rseq/param_test.c index 9869369a8607..f3687a90ec0c 100644 --- a/tools/testing/selftests/rseq/param_test.c +++ b/tools/testing/selftests/rseq/param_test.c @@ -16,6 +16,7 @@ #include #include #include +#include =20 static inline pid_t rseq_gettid(void) { @@ -36,7 +37,7 @@ static int opt_modulo, verbose; =20 static int opt_yield, opt_signal, opt_sleep, opt_disable_rseq, opt_threads =3D 200, - opt_disable_mod =3D 0, opt_test =3D 's', opt_mb =3D 0; + opt_disable_mod =3D 0, opt_test =3D 's'; =20 static long long opt_reps =3D 5000; =20 @@ -264,6 +265,63 @@ unsigned int yield_mod_cnt, nr_abort; =20 #include "rseq.h" =20 +static enum rseq_mo opt_mo =3D RSEQ_MO_RELAXED; + +#ifdef RSEQ_ARCH_HAS_OFFSET_DEREF_ADDV +#define TEST_MEMBARRIER + +static int sys_membarrier(int cmd, int flags, int cpu_id) +{ + return syscall(__NR_membarrier, cmd, flags, cpu_id); +} +#endif + +#ifdef BUILDOPT_RSEQ_PERCPU_VM_VCPU_ID +# define RSEQ_PERCPU RSEQ_PERCPU_VM_VCPU_ID +static +int get_current_cpu_id(void) +{ + return rseq_current_vm_vcpu_id(); +} +static +bool rseq_validate_cpu_id(void) +{ + return rseq_vm_vcpu_id_available(); +} +# ifdef TEST_MEMBARRIER +/* + * Membarrier does not currently support targeting a vm_vcpu_id, so + * issue the barrier on all cpus. + */ +static +int rseq_membarrier_expedited(int cpu) +{ + return sys_membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ, + 0, 0); +} +# endif /* TEST_MEMBARRIER */ +#else +# define RSEQ_PERCPU RSEQ_PERCPU_CPU_ID +static +int get_current_cpu_id(void) +{ + return rseq_cpu_start(); +} +static +bool rseq_validate_cpu_id(void) +{ + return rseq_current_cpu_raw() >=3D 0; +} +# ifdef TEST_MEMBARRIER +static +int rseq_membarrier_expedited(int cpu) +{ + return sys_membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ, + MEMBARRIER_CMD_FLAG_CPU, cpu); +} +# endif /* TEST_MEMBARRIER */ +#endif + struct percpu_lock_entry { intptr_t v; } __attribute__((aligned(128))); @@ -351,8 +409,9 @@ static int rseq_this_cpu_lock(struct percpu_lock *lock) for (;;) { int ret; =20 - cpu =3D rseq_cpu_start(); - ret =3D rseq_cmpeqv_storev(&lock->c[cpu].v, + cpu =3D get_current_cpu_id(); + ret =3D rseq_cmpeqv_storev(RSEQ_MO_RELAXED, RSEQ_PERCPU, + &lock->c[cpu].v, 0, 1, cpu); if (rseq_likely(!ret)) break; @@ -469,8 +528,9 @@ void *test_percpu_inc_thread(void *arg) do { int cpu; =20 - cpu =3D rseq_cpu_start(); - ret =3D rseq_addv(&data->c[cpu].count, 1, cpu); + cpu =3D get_current_cpu_id(); + ret =3D rseq_addv(RSEQ_MO_RELAXED, RSEQ_PERCPU, + &data->c[cpu].count, 1, cpu); } while (rseq_unlikely(ret)); #ifndef BENCHMARK if (i !=3D 0 && !(i % (reps / 10))) @@ -539,13 +599,14 @@ void this_cpu_list_push(struct percpu_list *list, intptr_t *targetptr, newval, expect; int ret; =20 - cpu =3D rseq_cpu_start(); + cpu =3D get_current_cpu_id(); /* Load list->c[cpu].head with single-copy atomicity. */ expect =3D (intptr_t)RSEQ_READ_ONCE(list->c[cpu].head); newval =3D (intptr_t)node; targetptr =3D (intptr_t *)&list->c[cpu].head; node->next =3D (struct percpu_list_node *)expect; - ret =3D rseq_cmpeqv_storev(targetptr, expect, newval, cpu); + ret =3D rseq_cmpeqv_storev(RSEQ_MO_RELAXED, RSEQ_PERCPU, + targetptr, expect, newval, cpu); if (rseq_likely(!ret)) break; /* Retry if comparison fails or rseq aborts. */ @@ -571,13 +632,14 @@ struct percpu_list_node *this_cpu_list_pop(struct per= cpu_list *list, long offset; int ret; =20 - cpu =3D rseq_cpu_start(); + cpu =3D get_current_cpu_id(); targetptr =3D (intptr_t *)&list->c[cpu].head; expectnot =3D (intptr_t)NULL; offset =3D offsetof(struct percpu_list_node, next); load =3D (intptr_t *)&head; - ret =3D rseq_cmpnev_storeoffp_load(targetptr, expectnot, - offset, load, cpu); + ret =3D rseq_cmpnev_storeoffp_load(RSEQ_MO_RELAXED, RSEQ_PERCPU, + targetptr, expectnot, + offset, load, cpu); if (rseq_likely(!ret)) { node =3D head; break; @@ -715,7 +777,7 @@ bool this_cpu_buffer_push(struct percpu_buffer *buffer, intptr_t offset; int ret; =20 - cpu =3D rseq_cpu_start(); + cpu =3D get_current_cpu_id(); offset =3D RSEQ_READ_ONCE(buffer->c[cpu].offset); if (offset =3D=3D buffer->c[cpu].buflen) break; @@ -723,14 +785,9 @@ bool this_cpu_buffer_push(struct percpu_buffer *buffer, targetptr_spec =3D (intptr_t *)&buffer->c[cpu].array[offset]; newval_final =3D offset + 1; targetptr_final =3D &buffer->c[cpu].offset; - if (opt_mb) - ret =3D rseq_cmpeqv_trystorev_storev_release( - targetptr_final, offset, targetptr_spec, - newval_spec, newval_final, cpu); - else - ret =3D rseq_cmpeqv_trystorev_storev(targetptr_final, - offset, targetptr_spec, newval_spec, - newval_final, cpu); + ret =3D rseq_cmpeqv_trystorev_storev(opt_mo, RSEQ_PERCPU, + targetptr_final, offset, targetptr_spec, + newval_spec, newval_final, cpu); if (rseq_likely(!ret)) { result =3D true; break; @@ -753,7 +810,7 @@ struct percpu_buffer_node *this_cpu_buffer_pop(struct p= ercpu_buffer *buffer, intptr_t offset; int ret; =20 - cpu =3D rseq_cpu_start(); + cpu =3D get_current_cpu_id(); /* Load offset with single-copy atomicity. */ offset =3D RSEQ_READ_ONCE(buffer->c[cpu].offset); if (offset =3D=3D 0) { @@ -763,7 +820,8 @@ struct percpu_buffer_node *this_cpu_buffer_pop(struct p= ercpu_buffer *buffer, head =3D RSEQ_READ_ONCE(buffer->c[cpu].array[offset - 1]); newval =3D offset - 1; targetptr =3D (intptr_t *)&buffer->c[cpu].offset; - ret =3D rseq_cmpeqv_cmpeqv_storev(targetptr, offset, + ret =3D rseq_cmpeqv_cmpeqv_storev(RSEQ_MO_RELAXED, RSEQ_PERCPU, + targetptr, offset, (intptr_t *)&buffer->c[cpu].array[offset - 1], (intptr_t)head, newval, cpu); if (rseq_likely(!ret)) @@ -920,7 +978,7 @@ bool this_cpu_memcpy_buffer_push(struct percpu_memcpy_b= uffer *buffer, size_t copylen; int ret; =20 - cpu =3D rseq_cpu_start(); + cpu =3D get_current_cpu_id(); /* Load offset with single-copy atomicity. */ offset =3D RSEQ_READ_ONCE(buffer->c[cpu].offset); if (offset =3D=3D buffer->c[cpu].buflen) @@ -931,15 +989,11 @@ bool this_cpu_memcpy_buffer_push(struct percpu_memcpy= _buffer *buffer, copylen =3D sizeof(item); newval_final =3D offset + 1; targetptr_final =3D &buffer->c[cpu].offset; - if (opt_mb) - ret =3D rseq_cmpeqv_trymemcpy_storev_release( - targetptr_final, offset, - destptr, srcptr, copylen, - newval_final, cpu); - else - ret =3D rseq_cmpeqv_trymemcpy_storev(targetptr_final, - offset, destptr, srcptr, copylen, - newval_final, cpu); + ret =3D rseq_cmpeqv_trymemcpy_storev( + opt_mo, RSEQ_PERCPU, + targetptr_final, offset, + destptr, srcptr, copylen, + newval_final, cpu); if (rseq_likely(!ret)) { result =3D true; break; @@ -964,7 +1018,7 @@ bool this_cpu_memcpy_buffer_pop(struct percpu_memcpy_b= uffer *buffer, size_t copylen; int ret; =20 - cpu =3D rseq_cpu_start(); + cpu =3D get_current_cpu_id(); /* Load offset with single-copy atomicity. */ offset =3D RSEQ_READ_ONCE(buffer->c[cpu].offset); if (offset =3D=3D 0) @@ -975,8 +1029,8 @@ bool this_cpu_memcpy_buffer_pop(struct percpu_memcpy_b= uffer *buffer, copylen =3D sizeof(*item); newval_final =3D offset - 1; targetptr_final =3D &buffer->c[cpu].offset; - ret =3D rseq_cmpeqv_trymemcpy_storev(targetptr_final, - offset, destptr, srcptr, copylen, + ret =3D rseq_cmpeqv_trymemcpy_storev(RSEQ_MO_RELAXED, RSEQ_PERCPU, + targetptr_final, offset, destptr, srcptr, copylen, newval_final, cpu); if (rseq_likely(!ret)) { result =3D true; @@ -1151,7 +1205,7 @@ static int set_signal_handler(void) } =20 /* Test MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU membarrier command. */ -#ifdef RSEQ_ARCH_HAS_OFFSET_DEREF_ADDV +#ifdef TEST_MEMBARRIER struct test_membarrier_thread_args { int stop; intptr_t percpu_list_ptr; @@ -1178,9 +1232,10 @@ void *test_membarrier_worker_thread(void *arg) int ret; =20 do { - int cpu =3D rseq_cpu_start(); + int cpu =3D get_current_cpu_id(); =20 - ret =3D rseq_offset_deref_addv(&args->percpu_list_ptr, + ret =3D rseq_offset_deref_addv(RSEQ_MO_RELAXED, RSEQ_PERCPU, + &args->percpu_list_ptr, sizeof(struct percpu_list_entry) * cpu, 1, cpu); } while (rseq_unlikely(ret)); } @@ -1217,11 +1272,6 @@ void test_membarrier_free_percpu_list(struct percpu_= list *list) free(list->c[i].head); } =20 -static int sys_membarrier(int cmd, int flags, int cpu_id) -{ - return syscall(__NR_membarrier, cmd, flags, cpu_id); -} - /* * The manager thread swaps per-cpu lists that worker threads see, * and validates that there are no unexpected modifications. @@ -1260,8 +1310,7 @@ void *test_membarrier_manager_thread(void *arg) =20 /* Make list_b "active". */ atomic_store(&args->percpu_list_ptr, (intptr_t)&list_b); - if (sys_membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ, - MEMBARRIER_CMD_FLAG_CPU, cpu_a) && + if (rseq_membarrier_expedited(cpu_a) && errno !=3D ENXIO /* missing CPU */) { perror("sys_membarrier"); abort(); @@ -1284,8 +1333,7 @@ void *test_membarrier_manager_thread(void *arg) =20 /* Make list_a "active". */ atomic_store(&args->percpu_list_ptr, (intptr_t)&list_a); - if (sys_membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ, - MEMBARRIER_CMD_FLAG_CPU, cpu_b) && + if (rseq_membarrier_expedited(cpu_b) && errno !=3D ENXIO /* missing CPU*/) { perror("sys_membarrier"); abort(); @@ -1356,7 +1404,7 @@ void test_membarrier(void) abort(); } } -#else /* RSEQ_ARCH_HAS_OFFSET_DEREF_ADDV */ +#else /* TEST_MEMBARRIER */ void test_membarrier(void) { fprintf(stderr, "rseq_offset_deref_addv is not implemented on this archit= ecture. " @@ -1513,7 +1561,7 @@ int main(int argc, char **argv) verbose =3D 1; break; case 'M': - opt_mb =3D 1; + opt_mo =3D RSEQ_MO_RELEASE; break; default: show_usage(argc, argv); @@ -1533,6 +1581,10 @@ int main(int argc, char **argv) =20 if (!opt_disable_rseq && rseq_register_current_thread()) goto error; + if (!opt_disable_rseq && !rseq_validate_cpu_id()) { + fprintf(stderr, "Error: cpu id getter unavailable\n"); + goto error; + } switch (opt_test) { case 's': printf_verbose("spinlock\n"); diff --git a/tools/testing/selftests/rseq/run_param_test.sh b/tools/testing= /selftests/rseq/run_param_test.sh index f51bc83c9e41..11b5424e8b78 100755 --- a/tools/testing/selftests/rseq/run_param_test.sh +++ b/tools/testing/selftests/rseq/run_param_test.sh @@ -42,6 +42,11 @@ function do_tests() ./param_test ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@} ${EXTRA_A= RGS} || exit 1 echo "Running compare-twice test ${TEST_NAME[$i]}" ./param_test_compare_twice ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} = ${@} ${EXTRA_ARGS} || exit 1 + + echo "Running vm vcpu_id test ${TEST_NAME[$i]}" + ./param_test_vm_vcpu_id ${TEST_LIST[$i]} -r ${REPS} -t ${NR_THREADS} ${@= } ${EXTRA_ARGS} || exit 1 + echo "Running vm vcpu_id compare-twice test ${TEST_NAME[$i]}" + ./param_test_vm_vcpu_id_compare_twice ${TEST_LIST[$i]} -r ${REPS} -t ${N= R_THREADS} ${@} ${EXTRA_ARGS} || exit 1 let "i++" done } --=20 2.17.1 From nobody Tue Apr 14 22:46:25 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C180C19F2A for ; Fri, 29 Jul 2022 19:04:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238831AbiG2TE0 (ORCPT ); Fri, 29 Jul 2022 15:04:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238793AbiG2TDo (ORCPT ); Fri, 29 Jul 2022 15:03:44 -0400 Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D7EF18AB2B; Fri, 29 Jul 2022 12:03:04 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 2EB28318B71; Fri, 29 Jul 2022 15:03:02 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id p5u6UTBqgqEu; Fri, 29 Jul 2022 15:03:01 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 95D64318DEA; Fri, 29 Jul 2022 15:02:52 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 95D64318DEA DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1659121372; bh=h9tSFFf+9NDh5w3UMA4ak85MvaSoM48KyykzPpV1Wwk=; h=From:To:Date:Message-Id; b=d/jbki2RQHZPrWpesTlzRrNNgMZBrKIzKpv2++G4PzMbs5lENwed1L8iUvlsxfcDG zlhrnUq+XKu1+nJyy+IDQA8wTYjRo4rPs47aJ4UZFoYbZmAINbeMNml0xvioLMjS+m Nx1p2MjpuIMyT24eLVKZnSQh4tIBnwr7o73gd5rtAcevi373DldB93aSipqHSuT4Gr QVyXILPF16HDY26/A5GtIMN3740asmcpBA/BkZfUE61kONCPEs24deuRQQ3h1L6sfc QPd/F0H+HPw/Ve+mdIScAMtK0kMuQLBER8BQQpVro0xKp2WIr4rz1bMA857Wz99zHP Kmeo0n+HSQ/Gg== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 4ViiKJQgwb0W; Fri, 29 Jul 2022 15:02:52 -0400 (EDT) Received: from localhost.localdomain (192-222-180-24.qc.cable.ebox.net [192.222.180.24]) by mail.efficios.com (Postfix) with ESMTPSA id 8A9B8318C74; Fri, 29 Jul 2022 15:02:48 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Thomas Gleixner , "Paul E . McKenney" , Boqun Feng , "H . Peter Anvin" , Paul Turner , linux-api@vger.kernel.org, Christian Brauner , Florian Weimer , David.Laight@ACULAB.COM, carlos@redhat.com, Peter Oskolkov , Mathieu Desnoyers Subject: [PATCH v3 22/23] selftests/rseq: x86: Implement rseq_load_u32_u32 Date: Fri, 29 Jul 2022 15:02:24 -0400 Message-Id: <20220729190225.12726-23-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> References: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Allow loading a pair of u32 within a rseq critical section. It can be used in situations where both rseq_abi()->vm_vcpu_id and rseq_abi()->node_id need to be sampled atomically with respect to preemption, signal delivery and migration. Signed-off-by: Mathieu Desnoyers --- tools/testing/selftests/rseq/rseq-x86-bits.h | 43 ++++++++++++++++++++ tools/testing/selftests/rseq/rseq.h | 14 +++++++ 2 files changed, 57 insertions(+) diff --git a/tools/testing/selftests/rseq/rseq-x86-bits.h b/tools/testing/s= elftests/rseq/rseq-x86-bits.h index 28ca77cc876c..ef961ab012e5 100644 --- a/tools/testing/selftests/rseq/rseq-x86-bits.h +++ b/tools/testing/selftests/rseq/rseq-x86-bits.h @@ -990,4 +990,47 @@ int RSEQ_TEMPLATE_IDENTIFIER(rseq_cmpeqv_trymemcpy_sto= rev)(intptr_t *v, intptr_t =20 #endif =20 +#if defined(RSEQ_TEMPLATE_CPU_ID_NONE) && defined(RSEQ_TEMPLATE_MO_RELAXED) + +#define RSEQ_ARCH_HAS_LOAD_U32_U32 + +static inline __attribute__((always_inline)) +int RSEQ_TEMPLATE_IDENTIFIER(rseq_load_u32_u32)(uint32_t *dst1, uint32_t *= src1, + uint32_t *dst2, uint32_t *src2) +{ + RSEQ_INJECT_C(9) + + __asm__ __volatile__ goto ( + RSEQ_ASM_DEFINE_TABLE(3, 1f, 2f, 4f) /* start, commit, abort */ + /* Start rseq by storing table entry pointer into rseq_cs. */ + RSEQ_ASM_STORE_RSEQ_CS(1, 3b, RSEQ_ASM_TP_SEGMENT:RSEQ_CS_OFFSET(%[rseq_= offset])) + RSEQ_INJECT_ASM(3) + "movl %[src1], %%eax\n\t" + "movl %%eax, %[dst1]\n\t" + "movl %[src2], %%eax\n\t" + "movl %%eax, %[dst2]\n\t" + "2:\n\t" + RSEQ_INJECT_ASM(4) + RSEQ_ASM_DEFINE_ABORT(4, "", abort) + : /* gcc asm goto does not allow outputs */ + : [rseq_offset] "r" (rseq_offset), + /* final store input */ + [dst1] "m" (*dst1), + [src1] "m" (*src1), + [dst2] "m" (*dst2), + [src2] "m" (*src2) + : "memory", "cc", "rax" + RSEQ_INJECT_CLOBBER + : abort + ); + rseq_after_asm_goto(); + return 0; +abort: + rseq_after_asm_goto(); + RSEQ_INJECT_FAILED + return -1; +} + +#endif /* defined(RSEQ_TEMPLATE_CPU_ID_NONE) && defined(RSEQ_TEMPLATE_MO_R= ELAXED) */ + #include "rseq-bits-reset.h" diff --git a/tools/testing/selftests/rseq/rseq.h b/tools/testing/selftests/= rseq/rseq.h index 95a76a1c3b27..30fa8bfd874e 100644 --- a/tools/testing/selftests/rseq/rseq.h +++ b/tools/testing/selftests/rseq/rseq.h @@ -381,4 +381,18 @@ int rseq_cmpeqv_trymemcpy_storev(enum rseq_mo rseq_mo,= enum rseq_percpu_mode per } } =20 +#ifdef RSEQ_ARCH_HAS_LOAD_U32_U32 + +static inline __attribute__((always_inline)) +int rseq_load_u32_u32(enum rseq_mo rseq_mo, + uint32_t *dst1, uint32_t *src1, + uint32_t *dst2, uint32_t *src2) +{ + if (rseq_mo !=3D RSEQ_MO_RELAXED) + return -1; + return rseq_load_u32_u32_relaxed(dst1, src1, dst2, src2); +} + +#endif + #endif /* RSEQ_H_ */ --=20 2.17.1 From nobody Tue Apr 14 22:46:25 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE573C00144 for ; Fri, 29 Jul 2022 19:04:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238383AbiG2TEe (ORCPT ); Fri, 29 Jul 2022 15:04:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51286 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238824AbiG2TDp (ORCPT ); Fri, 29 Jul 2022 15:03:45 -0400 Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 97BF68AB37; Fri, 29 Jul 2022 12:03:05 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 897C33190A9; Fri, 29 Jul 2022 15:03:02 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id FjSsyCqPWGBL; Fri, 29 Jul 2022 15:03:02 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id AC257318E28; Fri, 29 Jul 2022 15:02:52 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com AC257318E28 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1659121372; bh=GI5TUw70TsFggMy26+Gy48ykoT2fvjxXI2AaoplVknA=; h=From:To:Date:Message-Id; b=p99RvNp7adxUrvSbURY7tSPn34Iw1skMde3C1BfzXR8dlN6ewpUPzEvHIlXuhYgi3 2wHdtazZG/+0JcwO7SikdE64FfOJ9TWYJBD7xvt+nq0CnkN7DyMyBF/+klaMBGBPQm tIQ7rc/DA9NsjtYcdRhbLPBThGPslXhE7m+FKgoMP/A3/XFTKTkzIZsONlwIobVvel +PlqCSjVKPsY8BV7hDIHn7hOnvlZFNz7YFzoOsLRMb3O3a41zSuwy5/QhygMsPcIor DRp7OPmY62vg6JNIuniG1BUvc30emx9JmYNtM6KQoUuoAu0cCRMqwBfDWVISG071zT +gUyOBft4B3Qw== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id dSlcnOIRj8i1; Fri, 29 Jul 2022 15:02:52 -0400 (EDT) Received: from localhost.localdomain (192-222-180-24.qc.cable.ebox.net [192.222.180.24]) by mail.efficios.com (Postfix) with ESMTPSA id D7471318C7C; Fri, 29 Jul 2022 15:02:48 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Thomas Gleixner , "Paul E . McKenney" , Boqun Feng , "H . Peter Anvin" , Paul Turner , linux-api@vger.kernel.org, Christian Brauner , Florian Weimer , David.Laight@ACULAB.COM, carlos@redhat.com, Peter Oskolkov , Mathieu Desnoyers Subject: [PATCH v3 23/23] selftests/rseq: Implement numa node id vs vm_vcpu_id invariant test Date: Fri, 29 Jul 2022 15:02:25 -0400 Message-Id: <20220729190225.12726-24-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> References: <20220729190225.12726-1-mathieu.desnoyers@efficios.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" On all architectures except Power, the NUMA topology is never reconfigured after a CPU has been associated with a NUMA node in the system lifetime. Even on Power, we can assume that NUMA topology reconfiguration happens rarely, and therefore we do not expect it to happen while the NUMA test is running. This test validates that the mapping between a vm_vcpu_id and a numa node id remains valid for the process lifetime. In other words, it validates that if any thread within the process running on behalf of a vm_vcpu_id N observes a NUMA node id M, all threads within this process will always observe the same NUMA node id value when running on behalf of that same vm_vcpu_id. This characteristic is important for NUMA locality. This test is skipped on architectures that do not implement rseq_load_u32_u32. Signed-off-by: Mathieu Desnoyers --- tools/testing/selftests/rseq/.gitignore | 1 + tools/testing/selftests/rseq/Makefile | 2 +- .../testing/selftests/rseq/basic_numa_test.c | 117 ++++++++++++++++++ 3 files changed, 119 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/rseq/basic_numa_test.c diff --git a/tools/testing/selftests/rseq/.gitignore b/tools/testing/selfte= sts/rseq/.gitignore index db5c1a124c6c..9231abed69cc 100644 --- a/tools/testing/selftests/rseq/.gitignore +++ b/tools/testing/selftests/rseq/.gitignore @@ -1,4 +1,5 @@ # SPDX-License-Identifier: GPL-2.0-only +basic_numa_test basic_percpu_ops_test basic_percpu_ops_vm_vcpu_id_test basic_test diff --git a/tools/testing/selftests/rseq/Makefile b/tools/testing/selftest= s/rseq/Makefile index 3eec8e166385..4bf5b7202254 100644 --- a/tools/testing/selftests/rseq/Makefile +++ b/tools/testing/selftests/rseq/Makefile @@ -12,7 +12,7 @@ LDLIBS +=3D -lpthread -ldl # still track changes to header files and depend on shared object. OVERRIDE_TARGETS =3D 1 =20 -TEST_GEN_PROGS =3D basic_test basic_percpu_ops_test basic_percpu_ops_vm_vc= pu_id_test param_test \ +TEST_GEN_PROGS =3D basic_test basic_numa_test basic_percpu_ops_test basic_= percpu_ops_vm_vcpu_id_test param_test \ param_test_benchmark param_test_compare_twice param_test_vm_vcpu_id \ param_test_vm_vcpu_id_benchmark param_test_vm_vcpu_id_compare_twice =20 diff --git a/tools/testing/selftests/rseq/basic_numa_test.c b/tools/testing= /selftests/rseq/basic_numa_test.c new file mode 100644 index 000000000000..45cb714b135c --- /dev/null +++ b/tools/testing/selftests/rseq/basic_numa_test.c @@ -0,0 +1,117 @@ +// SPDX-License-Identifier: LGPL-2.1 +/* + * Basic rseq NUMA test. Validate that (vm_vcpu_id, numa_node_id) pairs are + * invariant. The only known scenario where this is untrue is on Power whi= ch + * can reconfigure the NUMA topology on CPU hotunplug/hotplug sequence. + */ + +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include + +#include "rseq.h" + +#define NR_LOOPS 100000000 +#define NR_THREADS 16 + +#ifdef RSEQ_ARCH_HAS_LOAD_U32_U32 + +static +int cpu_numa_id[CPU_SETSIZE]; + +static +void numa_id_init(void) +{ + int i; + + for (i =3D 0; i < CPU_SETSIZE; i++) + cpu_numa_id[i] =3D -1; +} + +static +void *test_thread(void *arg) +{ + int i; + + if (rseq_register_current_thread()) { + fprintf(stderr, "Error: rseq_register_current_thread(...) failed(%d): %s= \n", + errno, strerror(errno)); + abort(); + } + + for (i =3D 0; i < NR_LOOPS; i++) { + uint32_t vm_vcpu_id, node; + int cached_node_id; + + while (rseq_load_u32_u32(RSEQ_MO_RELAXED, &vm_vcpu_id, &rseq_get_abi()->= vm_vcpu_id, + &node, &rseq_get_abi()->node_id) !=3D 0) { + /* Retry. */ + } + cached_node_id =3D RSEQ_READ_ONCE(cpu_numa_id[vm_vcpu_id]); + if (cached_node_id =3D=3D -1) { + RSEQ_WRITE_ONCE(cpu_numa_id[vm_vcpu_id], node); + } else { + if (node !=3D cached_node_id) { + fprintf(stderr, "Error: NUMA node id discrepancy: vm_vcpu_id %u cached= node id %d node id %u.\n", + vm_vcpu_id, cached_node_id, node); + fprintf(stderr, "This is likely a kernel bug, or caused by a concurren= t NUMA topology reconfiguration.\n"); + abort(); + } + } + } + + if (rseq_unregister_current_thread()) { + fprintf(stderr, "Error: rseq_unregister_current_thread(...) failed(%d): = %s\n", + errno, strerror(errno)); + abort(); + } + return NULL; +} + +static +int test_numa(void) +{ + pthread_t tid[NR_THREADS]; + int err, i; + void *tret; + + numa_id_init(); + + printf("testing rseq (vm_vcpu_id, numa_node_id) invariant, single thread\= n"); + + (void) test_thread(NULL); + + printf("testing rseq (vm_vcpu_id, numa_node_id) invariant, multi-threaded= \n"); + + for (i =3D 0; i < NR_THREADS; i++) { + err =3D pthread_create(&tid[i], NULL, test_thread, NULL); + if (err !=3D 0) + abort(); + } + + for (i =3D 0; i < NR_THREADS; i++) { + err =3D pthread_join(tid[i], &tret); + if (err !=3D 0) + abort(); + } + + return 0; +} +#else +static +int test_numa(void) +{ + fprintf(stderr, "rseq_load_u32_u32 is not implemented on this architectur= e. " + "Skipping numa test.\n"); + return 0; +} +#endif + +int main(int argc, char **argv) +{ + return test_numa(); +} --=20 2.17.1