Masked user access optimizes the Spectre-V1 speculation barrier on
architectures which support it. As rseq_handle_notify_resume() is
frequently invoked, the access to the critical section pointer in the rseq
ABI is a hotpath, which is worth to optimize.
Replace also the clearing with the optimized version.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
---
kernel/rseq.c | 23 +++--------------------
1 file changed, 3 insertions(+), 20 deletions(-)
--- a/kernel/rseq.c
+++ b/kernel/rseq.c
@@ -245,20 +245,9 @@ static int rseq_reset_rseq_cpu_node_id(s
/*
* Get the user-space pointer value stored in the 'rseq_cs' field.
*/
-static int rseq_get_rseq_cs_ptr_val(struct rseq __user *rseq, u64 *rseq_cs)
+static inline int rseq_get_rseq_cs_ptr_val(struct rseq __user *rseq, u64 *rseq_cs)
{
- if (!rseq_cs)
- return -EFAULT;
-
-#ifdef CONFIG_64BIT
- if (get_user(*rseq_cs, &rseq->rseq_cs))
- return -EFAULT;
-#else
- if (copy_from_user(rseq_cs, &rseq->rseq_cs, sizeof(*rseq_cs)))
- return -EFAULT;
-#endif
-
- return 0;
+ return get_user_masked_u64(rseq_cs, &rseq->rseq_cs);
}
/*
@@ -358,13 +347,7 @@ static int clear_rseq_cs(struct rseq __u
*
* Set rseq_cs to NULL.
*/
-#ifdef CONFIG_64BIT
- return put_user(0UL, &rseq->rseq_cs);
-#else
- if (clear_user(&rseq->rseq_cs, sizeof(rseq->rseq_cs)))
- return -EFAULT;
- return 0;
-#endif
+ return put_user_masked_u64(0UL, &rseq->rseq_cs);
}
/*