[v1] sched: fix warning in sched_setaffinity

[PATCH] sched: fix warning in sched_setaffinity

Posted by Josh Don 1 year, 3 months ago

Commit 8f9ea86fdf99b added some logic to sched_setaffinity that included
a WARN when a per-task affinity assignment races with a cpuset update.

Specifically, we can have a race where a cpuset update results in the
task affinity no longer being a subset of the cpuset. That's fine; we
have a fallback to instead use the cpuset mask. However, we have a WARN
set up that will trigger if the cpuset mask has no overlap at all with
the requested task affinity. This shouldn't be a warning condition; its
trivial to create this condition.

Reproduced the warning by the following setup:

- $PID inside a cpuset cgroup
- another thread repeatedly switching the cpuset cpus from 1-2 to just 1
- another thread repeatedly setting the $PID affinity (via taskset) to 2

Fixes: 8f9ea86fdf99b ("sched: Always preserve the user requested cpumask")
Signed-off-by: Josh Don <joshdon@google.com>
---
 kernel/sched/syscalls.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index 4fae3cf25a3a..3a88f7c0cb69 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -1321,7 +1321,7 @@ int __sched_setaffinity(struct task_struct *p, struct affinity_context *ctx)
 			bool empty = !cpumask_and(new_mask, new_mask,
 						  ctx->user_mask);
 
-			if (WARN_ON_ONCE(empty))
+			if (empty)
 				cpumask_copy(new_mask, cpus_allowed);
 		}
 		__set_cpus_allowed_ptr(p, ctx);
-- 
2.46.0.469.g59c65b2a67-goog

Re: [PATCH] sched: fix warning in sched_setaffinity

Posted by Josh Don 1 year, 2 months ago

On Thu, Aug 29, 2024 at 3:04 PM Josh Don <joshdon@google.com> wrote:
>
> Commit 8f9ea86fdf99b added some logic to sched_setaffinity that included
> a WARN when a per-task affinity assignment races with a cpuset update.
>
> Specifically, we can have a race where a cpuset update results in the
> task affinity no longer being a subset of the cpuset. That's fine; we
> have a fallback to instead use the cpuset mask. However, we have a WARN
> set up that will trigger if the cpuset mask has no overlap at all with
> the requested task affinity. This shouldn't be a warning condition; its
> trivial to create this condition.
>
> Reproduced the warning by the following setup:
>
> - $PID inside a cpuset cgroup
> - another thread repeatedly switching the cpuset cpus from 1-2 to just 1
> - another thread repeatedly setting the $PID affinity (via taskset) to 2
>
> Fixes: 8f9ea86fdf99b ("sched: Always preserve the user requested cpumask")
> Signed-off-by: Josh Don <joshdon@google.com>

Gentle ping to bump this in case it got lost.

I've also collected the following:

Acked-by: Waiman Long <longman@redhat.com>
Tested-by: Madadi Vineeth Reddy <vineethr@linux.ibm.com>

Best,
Josh

Re: [PATCH] sched: fix warning in sched_setaffinity

Posted by Josh Don 1 year, 1 month ago

> Gentle ping to bump this in case it got lost.

Nudge :)

> I've also collected the following:
>
> Acked-by: Waiman Long <longman@redhat.com>
> Tested-by: Madadi Vineeth Reddy <vineethr@linux.ibm.com>

Also adding

Acked-and-tested-by: Vincent Guittot <vincent.guittot@linaro.org>

Best,
Josh

Re: [PATCH] sched: fix warning in sched_setaffinity

Posted by Vincent Guittot 1 year, 2 months ago

On Mon, 30 Sept 2024 at 22:26, Josh Don <joshdon@google.com> wrote:
>
> On Thu, Aug 29, 2024 at 3:04 PM Josh Don <joshdon@google.com> wrote:
> >
> > Commit 8f9ea86fdf99b added some logic to sched_setaffinity that included
> > a WARN when a per-task affinity assignment races with a cpuset update.
> >
> > Specifically, we can have a race where a cpuset update results in the
> > task affinity no longer being a subset of the cpuset. That's fine; we
> > have a fallback to instead use the cpuset mask. However, we have a WARN
> > set up that will trigger if the cpuset mask has no overlap at all with
> > the requested task affinity. This shouldn't be a warning condition; its
> > trivial to create this condition.
> >
> > Reproduced the warning by the following setup:
> >
> > - $PID inside a cpuset cgroup
> > - another thread repeatedly switching the cpuset cpus from 1-2 to just 1
> > - another thread repeatedly setting the $PID affinity (via taskset) to 2
> >
> > Fixes: 8f9ea86fdf99b ("sched: Always preserve the user requested cpumask")
> > Signed-off-by: Josh Don <joshdon@google.com>
>
> Gentle ping to bump this in case it got lost.

I have also been able to reproduce the race and the WARN with the
steps described in the commit description

Acked-and-tested-by: Vincent Guittot <vincent.guittot@linaro.org>

>
> I've also collected the following:
>
> Acked-by: Waiman Long <longman@redhat.com>
> Tested-by: Madadi Vineeth Reddy <vineethr@linux.ibm.com>
>
> Best,
> Josh

Re: [PATCH] sched: fix warning in sched_setaffinity

Posted by Madadi Vineeth Reddy 1 year, 3 months ago

Hi Josh Don,

On 30/08/24 03:34, Josh Don wrote:
> Commit 8f9ea86fdf99b added some logic to sched_setaffinity that included
> a WARN when a per-task affinity assignment races with a cpuset update.
> 
> Specifically, we can have a race where a cpuset update results in the
> task affinity no longer being a subset of the cpuset. That's fine; we
> have a fallback to instead use the cpuset mask. However, we have a WARN
> set up that will trigger if the cpuset mask has no overlap at all with
> the requested task affinity. This shouldn't be a warning condition; its
> trivial to create this condition.
> 
> Reproduced the warning by the following setup:
> 
> - $PID inside a cpuset cgroup
> - another thread repeatedly switching the cpuset cpus from 1-2 to just 1
> - another thread repeatedly setting the $PID affinity (via taskset) to 2
> 

I was testing the patch using the following two scripts run concurrently:

Script 1:
while true; do
    echo 1 > /sys/fs/cgroup/test_group/cpuset.cpus;
    echo 1-2 > /sys/fs/cgroup/test_group/cpuset.cpus;
done

Script 2:
while true; do
    sudo taskset -p 0x2 $$;
done

However, I am unable to trigger the warning in dmesg on the unpatched kernel.
I was expecting to see the warning as described, but it doesn't seem to appear.

Additionally, I also tried the following script to increase the chances of
triggering the race condition:

while true; do
    echo 1 > /sys/fs/cgroup/test_group/cpuset.cpus;
    sudo taskset -p 0x2 $$;
    sleep 0.1;
    echo 1-2 > /sys/fs/cgroup/test_group/cpuset.cpus;
done

Despite this, the warning still does not appear in dmesg.

Am I missing something in my testing approach, or is there a different setup
required to reproduce the issue?

Thanks and Regards
Madadi Vineeth Reddy

> Fixes: 8f9ea86fdf99b ("sched: Always preserve the user requested cpumask")
> Signed-off-by: Josh Don <joshdon@google.com>

Re: [PATCH] sched: fix warning in sched_setaffinity

Posted by Josh Don 1 year, 3 months ago

Hi Madadi,

On Sun, Sep 1, 2024 at 7:25 AM Madadi Vineeth Reddy
<vineethr@linux.ibm.com> wrote:
>
> Hi Josh Don,
>
> On 30/08/24 03:34, Josh Don wrote:
> > Commit 8f9ea86fdf99b added some logic to sched_setaffinity that included
> > a WARN when a per-task affinity assignment races with a cpuset update.
> >
> > Specifically, we can have a race where a cpuset update results in the
> > task affinity no longer being a subset of the cpuset. That's fine; we
> > have a fallback to instead use the cpuset mask. However, we have a WARN
> > set up that will trigger if the cpuset mask has no overlap at all with
> > the requested task affinity. This shouldn't be a warning condition; its
> > trivial to create this condition.
> >
> > Reproduced the warning by the following setup:
> >
> > - $PID inside a cpuset cgroup
> > - another thread repeatedly switching the cpuset cpus from 1-2 to just 1
> > - another thread repeatedly setting the $PID affinity (via taskset) to 2
> >
>
> I was testing the patch using the following two scripts run concurrently:
>
> Script 1:
> while true; do
>     echo 1 > /sys/fs/cgroup/test_group/cpuset.cpus;
>     echo 1-2 > /sys/fs/cgroup/test_group/cpuset.cpus;
> done
>
> Script 2:
> while true; do
>     sudo taskset -p 0x2 $$;
> done
>
> However, I am unable to trigger the warning in dmesg on the unpatched kernel.
> I was expecting to see the warning as described, but it doesn't seem to appear.
>
> Additionally, I also tried the following script to increase the chances of
> triggering the race condition:
>
> while true; do
>     echo 1 > /sys/fs/cgroup/test_group/cpuset.cpus;
>     sudo taskset -p 0x2 $$;
>     sleep 0.1;
>     echo 1-2 > /sys/fs/cgroup/test_group/cpuset.cpus;
> done
>
> Despite this, the warning still does not appear in dmesg.
>
> Am I missing something in my testing approach, or is there a different setup
> required to reproduce the issue?

taskset -p 0x2 $$ will affine to cpu 1 :)

I'd recommend using the '-c' arg to specify the mask as a cpulist, as
it is easier to validate.

taskset -c -p 2 $$

>
> Thanks and Regards
> Madadi Vineeth Reddy
>
> > Fixes: 8f9ea86fdf99b ("sched: Always preserve the user requested cpumask")
> > Signed-off-by: Josh Don <joshdon@google.com>
>

Best,
Josh

Re: [PATCH] sched: fix warning in sched_setaffinity

Posted by Madadi Vineeth Reddy 1 year, 3 months ago

Hi Josh Don,

On 04/09/24 03:03, Josh Don wrote:
> Hi Madadi,
> 
> On Sun, Sep 1, 2024 at 7:25 AM Madadi Vineeth Reddy
> <vineethr@linux.ibm.com> wrote:
>>
>> Hi Josh Don,
>>
>> On 30/08/24 03:34, Josh Don wrote:
>>> Commit 8f9ea86fdf99b added some logic to sched_setaffinity that included
>>> a WARN when a per-task affinity assignment races with a cpuset update.
>>>
>>> Specifically, we can have a race where a cpuset update results in the
>>> task affinity no longer being a subset of the cpuset. That's fine; we
>>> have a fallback to instead use the cpuset mask. However, we have a WARN
>>> set up that will trigger if the cpuset mask has no overlap at all with
>>> the requested task affinity. This shouldn't be a warning condition; its
>>> trivial to create this condition.
>>>
>>> Reproduced the warning by the following setup:
>>>
>>> - $PID inside a cpuset cgroup
>>> - another thread repeatedly switching the cpuset cpus from 1-2 to just 1
>>> - another thread repeatedly setting the $PID affinity (via taskset) to 2
>>>
>>
>> I was testing the patch using the following two scripts run concurrently:
>>
>> Script 1:
>> while true; do
>>     echo 1 > /sys/fs/cgroup/test_group/cpuset.cpus;
>>     echo 1-2 > /sys/fs/cgroup/test_group/cpuset.cpus;
>> done
>>
>> Script 2:
>> while true; do
>>     sudo taskset -p 0x2 $$;
>> done
>>
>> However, I am unable to trigger the warning in dmesg on the unpatched kernel.
>> I was expecting to see the warning as described, but it doesn't seem to appear.
>>
>> Additionally, I also tried the following script to increase the chances of
>> triggering the race condition:
>>
>> while true; do
>>     echo 1 > /sys/fs/cgroup/test_group/cpuset.cpus;
>>     sudo taskset -p 0x2 $$;
>>     sleep 0.1;
>>     echo 1-2 > /sys/fs/cgroup/test_group/cpuset.cpus;
>> done
>>
>> Despite this, the warning still does not appear in dmesg.
>>
>> Am I missing something in my testing approach, or is there a different setup
>> required to reproduce the issue?
> 
> taskset -p 0x2 $$ will affine to cpu 1 :)
> 
> I'd recommend using the '-c' arg to specify the mask as a cpulist, as
> it is easier to validate.
> 
> taskset -c -p 2 $$
> 

Thanks for the clarification.

Tested-by: Madadi Vineeth Reddy <vineethr@linux.ibm.com>

Thanks and Regards
Madadi Vineeth Reddy

>>
>> Thanks and Regards
>> Madadi Vineeth Reddy
>>
>>> Fixes: 8f9ea86fdf99b ("sched: Always preserve the user requested cpumask")
>>> Signed-off-by: Josh Don <joshdon@google.com>
>>
> 
> Best,
> Josh

Re: [PATCH] sched: fix warning in sched_setaffinity

Posted by Waiman Long 1 year, 3 months ago

On 8/29/24 18:04, Josh Don wrote:
> Commit 8f9ea86fdf99b added some logic to sched_setaffinity that included
> a WARN when a per-task affinity assignment races with a cpuset update.
>
> Specifically, we can have a race where a cpuset update results in the
> task affinity no longer being a subset of the cpuset. That's fine; we
> have a fallback to instead use the cpuset mask. However, we have a WARN
> set up that will trigger if the cpuset mask has no overlap at all with
> the requested task affinity. This shouldn't be a warning condition; its
> trivial to create this condition.
>
> Reproduced the warning by the following setup:
>
> - $PID inside a cpuset cgroup
> - another thread repeatedly switching the cpuset cpus from 1-2 to just 1
> - another thread repeatedly setting the $PID affinity (via taskset) to 2
>
> Fixes: 8f9ea86fdf99b ("sched: Always preserve the user requested cpumask")
> Signed-off-by: Josh Don <joshdon@google.com>
> ---
>   kernel/sched/syscalls.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
> index 4fae3cf25a3a..3a88f7c0cb69 100644
> --- a/kernel/sched/syscalls.c
> +++ b/kernel/sched/syscalls.c
> @@ -1321,7 +1321,7 @@ int __sched_setaffinity(struct task_struct *p, struct affinity_context *ctx)
>   			bool empty = !cpumask_and(new_mask, new_mask,
>   						  ctx->user_mask);
>   
> -			if (WARN_ON_ONCE(empty))
> +			if (empty)
>   				cpumask_copy(new_mask, cpus_allowed);
>   		}
>   		__set_cpus_allowed_ptr(p, ctx);

Taking out the WARN_ON_ONCE() should be fine.

Acked-by: Waiman Long <longman@redhat.com>