Provide the actual decision function, which decides whether a time slice
extension is granted in the exit to user mode path when NEED_RESCHED is
evaluated.

The decision is made in two stages. First an inline quick check to avoid
going into the actual decision function. This checks whether:

 #1 the functionality is enabled

 #2 the exit is a return from interrupt to user mode

 #3 any TIF bit, which causes extra work is set. That includes TIF_RSEQ,
    which means the task was already scheduled out.
 
The slow path, which implements the actual user space ABI, is invoked
when:

  A) #1 is true, #2 is true and #3 is false

     It checks whether user space requested a slice extension by setting
     the request bit in the rseq slice_ctrl field. If so, it grants the
     extension and stores the slice expiry time, so that the actual exit
     code can double check whether the slice is already exhausted before
     going back.

  B) #1 - #3 are true _and_ a slice extension was granted in a previous
     loop iteration

     In this case the grant is revoked.

In case that the user space access faults or invalid state is detected, the
task is terminated with SIGSEGV.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
---
V2: Provide an extra stub for the !RSEQ case - Prateek
---
 include/linux/rseq_entry.h |  108 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 108 insertions(+)

--- a/include/linux/rseq_entry.h
+++ b/include/linux/rseq_entry.h
@@ -41,6 +41,7 @@ DECLARE_PER_CPU(struct rseq_stats, rseq_
 #ifdef CONFIG_RSEQ
 #include <linux/jump_label.h>
 #include <linux/rseq.h>
+#include <linux/sched/signal.h>
 #include <linux/uaccess.h>
 
 #include <linux/tracepoint-defs.h>
@@ -108,10 +109,116 @@ static __always_inline void rseq_slice_c
 	t->rseq.slice.state.granted = false;
 }
 
+static __always_inline bool rseq_grant_slice_extension(bool work_pending)
+{
+	struct task_struct *curr = current;
+	struct rseq_slice_ctrl usr_ctrl;
+	union rseq_slice_state state;
+	struct rseq __user *rseq;
+
+	if (!rseq_slice_extension_enabled())
+		return false;
+
+	/* If not enabled or not a return from interrupt, nothing to do. */
+	state = curr->rseq.slice.state;
+	state.enabled &= curr->rseq.event.user_irq;
+	if (likely(!state.state))
+		return false;
+
+	rseq = curr->rseq.usrptr;
+	scoped_user_rw_access(rseq, efault) {
+
+		/*
+		 * Quick check conditions where a grant is not possible or
+		 * needs to be revoked.
+		 *
+		 *  1) Any TIF bit which needs to do extra work aside of
+		 *     rescheduling prevents a grant.
+		 *
+		 *  2) A previous rescheduling request resulted in a slice
+		 *     extension grant.
+		 */
+		if (unlikely(work_pending || state.granted)) {
+			/* Clear user control unconditionally. No point for checking */
+			unsafe_put_user(0U, &rseq->slice_ctrl.all, efault);
+			rseq_slice_clear_grant(curr);
+			return false;
+		}
+
+		unsafe_get_user(usr_ctrl.all, &rseq->slice_ctrl.all, efault);
+		if (likely(!(usr_ctrl.request)))
+			return false;
+
+		/* Grant the slice extention */
+		usr_ctrl.request = 0;
+		usr_ctrl.granted = 1;
+		unsafe_put_user(usr_ctrl.all, &rseq->slice_ctrl.all, efault);
+	}
+
+	rseq_stat_inc(rseq_stats.s_granted);
+
+	curr->rseq.slice.state.granted = true;
+	/* Store expiry time for arming the timer on the way out */
+	curr->rseq.slice.expires = data_race(rseq_slice_ext_nsecs) + ktime_get_mono_fast_ns();
+	/*
+	 * This is racy against a remote CPU setting TIF_NEED_RESCHED in
+	 * several ways:
+	 *
+	 * 1)
+	 *	CPU0			CPU1
+	 *	clear_tsk()
+	 *				set_tsk()
+	 *	clear_preempt()
+	 *				Raise scheduler IPI on CPU0
+	 *	--> IPI
+	 *	    fold_need_resched() -> Folds correctly
+	 * 2)
+	 *	CPU0			CPU1
+	 *				set_tsk()
+	 *	clear_tsk()
+	 *	clear_preempt()
+	 *				Raise scheduler IPI on CPU0
+	 *	--> IPI
+	 *	    fold_need_resched() <- NOOP as TIF_NEED_RESCHED is false
+	 *
+	 * #1 is not any different from a regular remote reschedule as it
+	 *    sets the previously not set bit and then raises the IPI which
+	 *    folds it into the preempt counter
+	 *
+	 * #2 is obviously incorrect from a scheduler POV, but it's not
+	 *    differently incorrect than the code below clearing the
+	 *    reschedule request with the safety net of the timer.
+	 *
+	 * The important part is that the clearing is protected against the
+	 * scheduler IPI and also against any other interrupt which might
+	 * end up waking up a task and setting the bits in the middle of
+	 * the operation:
+	 *
+	 *	clear_tsk()
+	 *	---> Interrupt
+	 *		wakeup_on_this_cpu()
+	 *		set_tsk()
+	 *		set_preempt()
+	 *	clear_preempt()
+	 *
+	 * which would be inconsistent state.
+	 */
+	scoped_guard(irq) {
+		clear_tsk_need_resched(curr);
+		clear_preempt_need_resched();
+	}
+	return true;
+
+efault:
+	force_sig(SIGSEGV);
+	return false;
+}
+
 #else /* CONFIG_RSEQ_SLICE_EXTENSION */
 static inline bool rseq_slice_extension_enabled(void) { return false; }
 static inline bool rseq_arm_slice_extension_timer(void) { return false; }
 static inline void rseq_slice_clear_grant(struct task_struct *t) { }
+static inline bool rseq_grant_slice_extension(bool work_pending) { return false; }
 #endif /* !CONFIG_RSEQ_SLICE_EXTENSION */
 
 bool rseq_debug_update_user_cs(struct task_struct *t, struct pt_regs *regs, unsigned long csaddr);
@@ -646,6 +753,7 @@ static inline bool rseq_exit_to_user_mod
 static inline void rseq_syscall_exit_to_user_mode(void) { }
 static inline void rseq_irqentry_exit_to_user_mode(void) { }
 static inline void rseq_debug_syscall_return(struct pt_regs *regs) { }
+static inline bool rseq_grant_slice_extension(bool work_pending) { return false; }
 #endif /* !CONFIG_RSEQ */
 
 #endif /* _LINUX_RSEQ_ENTRY_H */

On Wed, 29 Oct 2025 14:22:30 +0100 (CET)
Thomas Gleixner <tglx@linutronix.de> wrote:

> +static __always_inline bool rseq_grant_slice_extension(bool work_pending)
> +{
> +	struct task_struct *curr = current;
> +	struct rseq_slice_ctrl usr_ctrl;
> +	union rseq_slice_state state;
> +	struct rseq __user *rseq;
> +
> +	if (!rseq_slice_extension_enabled())
> +		return false;
> +
> +	/* If not enabled or not a return from interrupt, nothing to do. */
> +	state = curr->rseq.slice.state;
> +	state.enabled &= curr->rseq.event.user_irq;
> +	if (likely(!state.state))
> +		return false;
> +
> +	rseq = curr->rseq.usrptr;
> +	scoped_user_rw_access(rseq, efault) {
> +
> +		/*
> +		 * Quick check conditions where a grant is not possible or
> +		 * needs to be revoked.
> +		 *
> +		 *  1) Any TIF bit which needs to do extra work aside of
> +		 *     rescheduling prevents a grant.
> +		 *

I'm curious to why any other TIF bit causes this to refuse a grant?

If deferred unwinding gets implemented, and profiling is enabled, it uses
task_work. From my understanding, task_work will set a TIF bit. Would this
mean that we would not be able to profile this feature with the deferred
unwinder? As profiling it will prevent it from being used?

-- Steve


> +		 *  2) A previous rescheduling request resulted in a slice
> +		 *     extension grant.
> +		 */
> +		if (unlikely(work_pending || state.granted)) {
> +			/* Clear user control unconditionally. No point for checking */
> +			unsafe_put_user(0U, &rseq->slice_ctrl.all, efault);
> +			rseq_slice_clear_grant(curr);
> +			return false;
> +		}
> +
> +		unsafe_get_user(usr_ctrl.all, &rseq->slice_ctrl.all, efault);
> +		if (likely(!(usr_ctrl.request)))
> +			return false;
> +
> +		/* Grant the slice extention */
> +		usr_ctrl.request = 0;
> +		usr_ctrl.granted = 1;
> +		unsafe_put_user(usr_ctrl.all, &rseq->slice_ctrl.all, efault);
> +	}
> +

On Wed, Oct 29 2025 at 16:08, Steven Rostedt wrote:
> On Wed, 29 Oct 2025 14:22:30 +0100 (CET)
> Thomas Gleixner <tglx@linutronix.de> wrote:
>> +		/*
>> +		 * Quick check conditions where a grant is not possible or
>> +		 * needs to be revoked.
>> +		 *
>> +		 *  1) Any TIF bit which needs to do extra work aside of
>> +		 *     rescheduling prevents a grant.
>> +		 *
>
> I'm curious to why any other TIF bit causes this to refuse a grant?
>
> If deferred unwinding gets implemented, and profiling is enabled, it uses
> task_work. From my understanding, task_work will set a TIF bit. Would this
> mean that we would not be able to profile this feature with the deferred
> unwinder? As profiling it will prevent it from being used?

You still can use it. The point is that a set TIF bit will do extra
work, which means extra scheduling latency. The extra work might be
short enough to still make the grant useful, but that's something which
needs to be worked out and analyzed. Quite some of the TIF bits actually
end up with another reschedule request.

As this whole thing is an opportunistic poor mans priority ceiling
attempt, I opted for the simple decision of not granting it when other
TIF bits are set. KISS rules :)

That's not set in stone and has no user space ABI relevance because it's
solely a kernel implementation detail.

> -- Steve

Can you please trim your replies as anybody else does?

Thanks,

        tglx

On Wed, 29 Oct 2025 22:46:12 +0100
Thomas Gleixner <tglx@linutronix.de> wrote:

> Can you please trim your replies as anybody else does?

I did trim it. I only kept the function in question, but deleted everything
else.

I do like to keep a bit of context, as sometimes I find people tend to trim
a bit too much.

-- Steve

On Wed, Oct 29 2025 at 18:04, Steven Rostedt wrote:

> On Wed, 29 Oct 2025 22:46:12 +0100
> Thomas Gleixner <tglx@linutronix.de> wrote:
>
>> Can you please trim your replies as anybody else does?
>
> I did trim it. I only kept the function in question, but deleted everything
> else.
>
> I do like to keep a bit of context, as sometimes I find people tend to trim
> a bit too much.

That's fine, but leaving stale quotes after

       Steve

is not.