From nobody Tue Jun 23 03:14:49 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7BDDEC433EF for ; Fri, 11 Mar 2022 16:14:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350168AbiCKQPX (ORCPT ); Fri, 11 Mar 2022 11:15:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38452 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350151AbiCKQPT (ORCPT ); Fri, 11 Mar 2022 11:15:19 -0500 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CB98818A798 for ; Fri, 11 Mar 2022 08:14:14 -0800 (PST) Received: by mail-wr1-x436.google.com with SMTP id u1so13646416wrg.11 for ; Fri, 11 Mar 2022 08:14:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=+DjhWNCi1uAUKSxL3o4kQa/u0xQ6TtRjRzuu8WmiLaA=; b=SL+WZJnQKpD0l7IOGXck34pr1hW6R496L7GYyFu03/z798wBhFlB76Bu6IUwjmvZfj lJ3anW+iAlNt8KPB6/NbTRTwT+rZEPi3Rae1oXXiV/KNNzbB86EfqLn83VSGRO3VNLra yhkX7jlBCKuaYv8qQe4U5enUppkW7wT/7Elbtaw68HFwpYz2VSCwS3aYh5fDHdfG6bAm 9MnLglDvVLr6nxBRefcerGlP+KjTJwGAA0WaHy+Cn6raBL3Y4a4j4EcALzv4/QgxuH12 Jpbq4/4LGg2XJm5slMrBm5X4cyQXoo9os/bpa9uy5Qn+gxmr1gCukxRWw8Cn+oybayDY UPXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=+DjhWNCi1uAUKSxL3o4kQa/u0xQ6TtRjRzuu8WmiLaA=; b=JH/mUnYZLG8tQhwP9dpc+vzXDQ5f6lZcIB7zgyC4Kza/8yYd3uIizG/mX1/M68Sm8R aZ0Cu0n20tqHvmComiZRnagNA+i+xPWeRAqlCD33VTxM9uwM3t3z4H4tRFYey8DWcppG Gc9TYefKHzlO1s25OBn7jEpdRAyl5Z2a2kVCK+dbrAEGJ7d9dxFUhQKToKbXEZCs07ag fPR4siFo9v97bD6HAkfsQkynXSL23Cuw2XaAaAwdCBeiODRiNvrUQ5ffJVGStu9xQ8hT +W+qeECF4pt3NbnLqT1U7gun+ThtW2BPXqJ7cNFizL6zsStYAakqCqjV4Y6pfG5CyGEd 4avg== X-Gm-Message-State: AOAM530SQbIXwkDWjDgcgUPBdzd7xx/CV8Emo8Sf/3l4XefR4J2GOYgk Ke7B41RM8TT3+HjVn/BfOXwAJA== X-Google-Smtp-Source: ABdhPJyb3fS6K92KyiMuBYFrxj8LkQPEJS4vXwEepSqukiGFEZ2HXo05KkhjqUq9oZ91TVJ8ZARRUw== X-Received: by 2002:a5d:500e:0:b0:1f0:2509:ee3f with SMTP id e14-20020a5d500e000000b001f02509ee3fmr8180955wrt.381.1647015253226; Fri, 11 Mar 2022 08:14:13 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:f:6020:70d9:405c:a1e4:4d23]) by smtp.gmail.com with ESMTPSA id 4-20020a056000154400b00203812ca383sm6464137wry.78.2022.03.11.08.14.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Mar 2022 08:14:12 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qais.yousef@arm.com, chris.hyser@oracle.com, pkondeti@codeaurora.org, valentin.schneider@arm.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, dhaval.giani@oracle.com, qperret@google.com, tim.c.chen@linux.intel.com, Vincent Guittot Subject: [PATCH 1/6] sched: Introduce latency-nice as a per-task attribute Date: Fri, 11 Mar 2022 17:14:01 +0100 Message-Id: <20220311161406.23497-2-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220311161406.23497-1-vincent.guittot@linaro.org> References: <20220311161406.23497-1-vincent.guittot@linaro.org> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Parth Shah Latency-nice indicates the latency requirements of a task with respect to the other tasks in the system. The value of the attribute can be within the range of [-20, 19] both inclusive to be in-line with the values just like task nice values. latency_nice =3D -20 indicates the task to have the least latency as compared to the tasks having latency_nice =3D +19. The latency_nice may affect only the CFS SCHED_CLASS by getting latency requirements from the userspace. Additionally, add debugging bits for newly added latency_nice attribute. Signed-off-by: Parth Shah [rebase] Signed-off-by: Vincent Guittot --- include/linux/sched.h | 1 + kernel/sched/debug.c | 1 + kernel/sched/sched.h | 18 ++++++++++++++++++ 3 files changed, 20 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index 508b91d57470..2aa889a59054 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -779,6 +779,7 @@ struct task_struct { int static_prio; int normal_prio; unsigned int rt_priority; + int latency_nice; =20 struct sched_entity se; struct sched_rt_entity rt; diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c index 102d6f70e84d..5d76a8927888 100644 --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -1043,6 +1043,7 @@ void proc_sched_show_task(struct task_struct *p, stru= ct pid_namespace *ns, #endif P(policy); P(prio); + P(latency_nice); if (task_has_dl_policy(p)) { P(dl.runtime); P(dl.deadline); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 9b33ba9c3c42..456ad2159eb1 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -105,6 +105,24 @@ extern void call_trace_sched_update_nr_running(struct = rq *rq, int count); */ #define NS_TO_JIFFIES(TIME) ((unsigned long)(TIME) / (NSEC_PER_SEC / HZ)) =20 +/* + * Latency nice is meant to provide scheduler hints about the relative + * latency requirements of a task with respect to other tasks. + * Thus a task with latency_nice =3D=3D 19 can be hinted as the task with = no + * latency requirements, in contrast to the task with latency_nice =3D=3D = -20 + * which should be given priority in terms of lower latency. + */ +#define MAX_LATENCY_NICE 19 +#define MIN_LATENCY_NICE -20 + +#define LATENCY_NICE_WIDTH \ + (MAX_LATENCY_NICE - MIN_LATENCY_NICE + 1) + +/* + * Default tasks should be treated as a task with latency_nice =3D 0. + */ +#define DEFAULT_LATENCY_NICE 0 + /* * Increase resolution of nice-level calculations for 64-bit architectures. * The extra resolution improves shares distribution and load balancing of --=20 2.17.1 From nobody Tue Jun 23 03:14:49 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB8E8C433EF for ; Fri, 11 Mar 2022 16:14:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350181AbiCKQP1 (ORCPT ); Fri, 11 Mar 2022 11:15:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38456 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350152AbiCKQPU (ORCPT ); Fri, 11 Mar 2022 11:15:20 -0500 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5926E191439 for ; Fri, 11 Mar 2022 08:14:16 -0800 (PST) Received: by mail-wr1-x42f.google.com with SMTP id r10so13694826wrp.3 for ; Fri, 11 Mar 2022 08:14:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=9NTMzRLoGwhdmZLI0ObjR+XF+Hbh3xQ8VsF4xT8xykA=; b=LsnZVgmhh5uS1QHpj2P4oaLfREx9Of1I8baPHD79rDOuDC9Pk4zK2cnXFgW+qJ+yqU aTcnryb/9nK684ESqGZxRLI3VQqImxz4vcM5e6jN2M7D1lbgci5PAdRMYfxoxzqhJRvw rjNrLUYwfNHjF1gudsP1xtqJ021ftJKnMRagij/zVzxJyCD8QTfGFQ65ZhyQNj6gjAVi V1a7Q5KPvl+RSE/Z25b98ilQQTjIjCGotvQx1Y5V3CPYmmxC0uSdyeRlmyc9ReO3qMVU L6JzgxIwDiQFDWyL1AKc3/Nv/eGQI2f/iplUT8tkE8o9MhchCb0yCl5A6xIi+QmS+SbO XyLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=9NTMzRLoGwhdmZLI0ObjR+XF+Hbh3xQ8VsF4xT8xykA=; b=qL1OcvkzLZOK9RBI3edJkLdyUWLoL+XGyCT2SlIuFk5/PUblRB6SmcJxTz+9w8Hbh3 WTSKQEcZjuF8JW8VL2TT+Yq7w6CpPhirxtrRhCK2sTfqFL1xVU3NIu7lMn4kDahPtstm wGWLDwj5onoRblmI6bH+ACprMbQ3WWHgUPquMvQw6C5Jv42+phb2MasLoXZveUuXX/X9 DyZMv76fUfAAGHJ2ZnlAnmfKuxNobbWieH0m6dqTX1DF0sukRJEPhQwl34yY23XRDkMB GD3Zm8evNf5R8gqligrvo2yb14h+tpmEpZI2VRxR/5zHlKBnCw1SVaIU9zUHb1MvZsyf /TMQ== X-Gm-Message-State: AOAM533T0I0Jif6a5iVFQuBgFkz9BTbk6xOjo+lVmrZvwN9uEd5OtHY0 VzBih8OKFgwltDiL+F5yfIWxTw== X-Google-Smtp-Source: ABdhPJxcbEF6e5XH+u6g5S39oS3QJ3F6/VpKABVcAM0aJiAwE5m8TIVhVRUDsVEZwJ8ONP/PAfSicA== X-Received: by 2002:a5d:678f:0:b0:1f0:2471:5a93 with SMTP id v15-20020a5d678f000000b001f024715a93mr8031486wru.164.1647015254891; Fri, 11 Mar 2022 08:14:14 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:f:6020:70d9:405c:a1e4:4d23]) by smtp.gmail.com with ESMTPSA id 4-20020a056000154400b00203812ca383sm6464137wry.78.2022.03.11.08.14.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Mar 2022 08:14:14 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qais.yousef@arm.com, chris.hyser@oracle.com, pkondeti@codeaurora.org, valentin.schneider@arm.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, dhaval.giani@oracle.com, qperret@google.com, tim.c.chen@linux.intel.com, Vincent Guittot Subject: [PATCH 2/6] sched/core: Propagate parent task's latency requirements to the child task Date: Fri, 11 Mar 2022 17:14:02 +0100 Message-Id: <20220311161406.23497-3-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220311161406.23497-1-vincent.guittot@linaro.org> References: <20220311161406.23497-1-vincent.guittot@linaro.org> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Parth Shah Clone parent task's latency_nice attribute to the forked child task. Reset the latency_nice value to default value when the child task is set to sched_reset_on_fork. Also, initialize init_task.latency_nice value with DEFAULT_LATENCY_NICE value Signed-off-by: Parth Shah [rebase] Signed-off-by: Vincent Guittot --- init/init_task.c | 1 + kernel/sched/core.c | 4 ++++ 2 files changed, 5 insertions(+) diff --git a/init/init_task.c b/init/init_task.c index 73cc8f03511a..2afa249c253b 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -78,6 +78,7 @@ struct task_struct init_task .prio =3D MAX_PRIO - 20, .static_prio =3D MAX_PRIO - 20, .normal_prio =3D MAX_PRIO - 20, + .latency_nice =3D 0, .policy =3D SCHED_NORMAL, .cpus_ptr =3D &init_task.cpus_mask, .user_cpus_ptr =3D NULL, diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 1d863d7f6ad7..157eef880d1d 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4393,6 +4393,9 @@ int sched_fork(unsigned long clone_flags, struct task= _struct *p) */ p->prio =3D current->normal_prio; =20 + /* Propagate the parent's latency requirements to the child as well */ + p->latency_nice =3D current->latency_nice; + uclamp_fork(p); =20 /* @@ -4409,6 +4412,7 @@ int sched_fork(unsigned long clone_flags, struct task= _struct *p) p->prio =3D p->normal_prio =3D p->static_prio; set_load_weight(p, false); =20 + p->latency_nice =3D DEFAULT_LATENCY_NICE; /* * We don't need the reset flag anymore after the fork. It has * fulfilled its duty: --=20 2.17.1 From nobody Tue Jun 23 03:14:49 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAFCAC433F5 for ; Fri, 11 Mar 2022 16:14:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350184AbiCKQPg (ORCPT ); Fri, 11 Mar 2022 11:15:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38610 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350154AbiCKQPW (ORCPT ); Fri, 11 Mar 2022 11:15:22 -0500 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B0DC2188861 for ; Fri, 11 Mar 2022 08:14:18 -0800 (PST) Received: by mail-wm1-x330.google.com with SMTP id k8-20020a05600c1c8800b003899c7ac55dso5920826wms.1 for ; Fri, 11 Mar 2022 08:14:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=0fsP3DO6qEug2CZV2Rozajvdffjtp4rs82zcMXC5MzU=; b=jYSCxMOgOyBjwOjzHvqFUJmoXxn++ufnpsdxt93OinSWe8HE5yTH+y76KA26mAXNZF QpT4zLOGpoKQBl1/VGF4IzxRRQolJC8WJqwNneOzYBUQz+SWoyrq6xUXmYLjkgbJH8LR PhvEPPFVUpd7vFRS/uV67SO1D3rQqRDK42xSaQacnrFLgnCN7sxiUB6en2qaZY4zrqFh 9Jd36H/wJXPbFiVqMInkbZShUgZqYnBOtov4ej1ndfbCFpgjafHUI3Om/zyD8IaY+5vc efKSAZCdFv2TrdkoSnXomwXDNR15V3pl2jnHtiy064h3spP44rm/nBIweB4fWUHSd2/4 oTLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=0fsP3DO6qEug2CZV2Rozajvdffjtp4rs82zcMXC5MzU=; b=ps58wog+8T/rVHEfUJyEWqiOD/1Wwosegrnn5NKf+Q0ppWJVJcVOu4RBC/YI9kPiBN 2XP+cMTfkBfjNu8ltORRic5AXh3RDrbBISozcqNy+lyPKfAhjAUIgwiCLQY7SLgGFIaa ToADPWosp3OpD1sJ6gDJrLAEuFUoB7Yov34e7a6RLgIpOm1VjPSSnTG4MdFOcXpIaHI8 XwyWT0UB4kF5N2xiV0d9usUepDiza1S7LHELiPHkX1a2rDnwOef5w9gZv17k37qMG6nK /wX87oFH8WikYc+ICf72b5WCy/t/q+2EK903aAk6r9oNiVXMTZhBTT10NBkHd3HAmYH/ rV+g== X-Gm-Message-State: AOAM5339CswjOchR3shxpebaO0+d0D0V7PC52Z56Mg7EL3Z+6QTfjKrV /TW1rhChakUnPLIRDRLRK6iKmg== X-Google-Smtp-Source: ABdhPJwhWx3LwiFdTcMMaCokFsSLpsM42EhBXcBR6CyF1h6KqCWOgRi7A96jxMKq//S9HTua+eNoxw== X-Received: by 2002:a7b:c3d5:0:b0:389:a49f:c7e6 with SMTP id t21-20020a7bc3d5000000b00389a49fc7e6mr16156212wmj.99.1647015257151; Fri, 11 Mar 2022 08:14:17 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:f:6020:70d9:405c:a1e4:4d23]) by smtp.gmail.com with ESMTPSA id 4-20020a056000154400b00203812ca383sm6464137wry.78.2022.03.11.08.14.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Mar 2022 08:14:16 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qais.yousef@arm.com, chris.hyser@oracle.com, pkondeti@codeaurora.org, valentin.schneider@arm.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, dhaval.giani@oracle.com, qperret@google.com, tim.c.chen@linux.intel.com, Vincent Guittot Subject: [PATCH 3/6] sched: Allow sched_{get,set}attr to change latency_nice of the task Date: Fri, 11 Mar 2022 17:14:03 +0100 Message-Id: <20220311161406.23497-4-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220311161406.23497-1-vincent.guittot@linaro.org> References: <20220311161406.23497-1-vincent.guittot@linaro.org> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Parth Shah Introduce the latency_nice attribute to sched_attr and provide a mechanism to change the value with the use of sched_setattr/sched_getattr syscall. Also add new flag "SCHED_FLAG_LATENCY_NICE" to hint the change in latency_nice of the task on every sched_setattr syscall. Signed-off-by: Parth Shah [rebase and add a dedicated __setscheduler_latency ] Signed-off-by: Vincent Guittot --- include/uapi/linux/sched.h | 4 +++- include/uapi/linux/sched/types.h | 19 +++++++++++++++++++ kernel/sched/core.c | 26 ++++++++++++++++++++++++++ tools/include/uapi/linux/sched.h | 4 +++- 4 files changed, 51 insertions(+), 2 deletions(-) diff --git a/include/uapi/linux/sched.h b/include/uapi/linux/sched.h index 3bac0a8ceab2..b2e932c25be6 100644 --- a/include/uapi/linux/sched.h +++ b/include/uapi/linux/sched.h @@ -132,6 +132,7 @@ struct clone_args { #define SCHED_FLAG_KEEP_PARAMS 0x10 #define SCHED_FLAG_UTIL_CLAMP_MIN 0x20 #define SCHED_FLAG_UTIL_CLAMP_MAX 0x40 +#define SCHED_FLAG_LATENCY_NICE 0x80 =20 #define SCHED_FLAG_KEEP_ALL (SCHED_FLAG_KEEP_POLICY | \ SCHED_FLAG_KEEP_PARAMS) @@ -143,6 +144,7 @@ struct clone_args { SCHED_FLAG_RECLAIM | \ SCHED_FLAG_DL_OVERRUN | \ SCHED_FLAG_KEEP_ALL | \ - SCHED_FLAG_UTIL_CLAMP) + SCHED_FLAG_UTIL_CLAMP | \ + SCHED_FLAG_LATENCY_NICE) =20 #endif /* _UAPI_LINUX_SCHED_H */ diff --git a/include/uapi/linux/sched/types.h b/include/uapi/linux/sched/ty= pes.h index f2c4589d4dbf..0aa4e3b6ed59 100644 --- a/include/uapi/linux/sched/types.h +++ b/include/uapi/linux/sched/types.h @@ -10,6 +10,7 @@ struct sched_param { =20 #define SCHED_ATTR_SIZE_VER0 48 /* sizeof first published struct */ #define SCHED_ATTR_SIZE_VER1 56 /* add: util_{min,max} */ +#define SCHED_ATTR_SIZE_VER2 60 /* add: latency_nice */ =20 /* * Extended scheduling parameters data structure. @@ -98,6 +99,22 @@ struct sched_param { * scheduled on a CPU with no more capacity than the specified value. * * A task utilization boundary can be reset by setting the attribute to -1. + * + * Latency Tolerance Attributes + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D + * + * A subset of sched_attr attributes allows to specify the relative latency + * requirements of a task with respect to the other tasks running/queued i= n the + * system. + * + * @ sched_latency_nice task's latency_nice value + * + * The latency_nice of a task can have any value in a range of + * [LATENCY_NICE_MIN..LATENCY_NICE_MAX]. + * + * A task with latency_nice with the value of LATENCY_NICE_MIN can be + * taken for a task with lower latency requirements as opposed to the task= with + * higher latency_nice. */ struct sched_attr { __u32 size; @@ -120,6 +137,8 @@ struct sched_attr { __u32 sched_util_min; __u32 sched_util_max; =20 + /* latency requirement hints */ + __s32 sched_latency_nice; }; =20 #endif /* _UAPI_LINUX_SCHED_TYPES_H */ diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 157eef880d1d..3edba1a38ecb 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -7219,6 +7219,16 @@ static void __setscheduler_params(struct task_struct= *p, p->rt_priority =3D attr->sched_priority; p->normal_prio =3D normal_prio(p); set_load_weight(p, true); + +} + +static void __setscheduler_latency(struct task_struct *p, + const struct sched_attr *attr) +{ + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) { + p->latency_prio =3D NICE_TO_LATENCY(attr->sched_latency_nice); + set_latency_weight(p); + } } =20 /* @@ -7345,6 +7355,13 @@ static int __sched_setscheduler(struct task_struct *= p, return retval; } =20 + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) { + if (attr->sched_latency_nice > MAX_LATENCY_NICE) + return -EINVAL; + if (attr->sched_latency_nice < MIN_LATENCY_NICE) + return -EINVAL; + } + if (pi) cpuset_read_lock(); =20 @@ -7379,6 +7396,9 @@ static int __sched_setscheduler(struct task_struct *p, goto change; if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP) goto change; + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE && + attr->sched_latency_nice !=3D p->latency_nice) + goto change; =20 p->sched_reset_on_fork =3D reset_on_fork; retval =3D 0; @@ -7467,6 +7487,7 @@ static int __sched_setscheduler(struct task_struct *p, __setscheduler_params(p, attr); __setscheduler_prio(p, newprio); } + __setscheduler_latency(p, attr); __setscheduler_uclamp(p, attr); =20 if (queued) { @@ -7677,6 +7698,9 @@ static int sched_copy_attr(struct sched_attr __user *= uattr, struct sched_attr *a size < SCHED_ATTR_SIZE_VER1) return -EINVAL; =20 + if ((attr->sched_flags & SCHED_FLAG_LATENCY_NICE) && + size < SCHED_ATTR_SIZE_VER2) + return -EINVAL; /* * XXX: Do we want to be lenient like existing syscalls; or do we want * to be strict and return an error on out-of-bounds values? @@ -7914,6 +7938,8 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sch= ed_attr __user *, uattr, get_params(p, &kattr); kattr.sched_flags &=3D SCHED_FLAG_ALL; =20 + kattr.sched_latency_nice =3D p->latency_nice; + #ifdef CONFIG_UCLAMP_TASK /* * This could race with another potential updater, but this is fine diff --git a/tools/include/uapi/linux/sched.h b/tools/include/uapi/linux/sc= hed.h index 3bac0a8ceab2..ecc4884bfe4b 100644 --- a/tools/include/uapi/linux/sched.h +++ b/tools/include/uapi/linux/sched.h @@ -132,6 +132,7 @@ struct clone_args { #define SCHED_FLAG_KEEP_PARAMS 0x10 #define SCHED_FLAG_UTIL_CLAMP_MIN 0x20 #define SCHED_FLAG_UTIL_CLAMP_MAX 0x40 +#define SCHED_FLAG_LATENCY_NICE 0X80 =20 #define SCHED_FLAG_KEEP_ALL (SCHED_FLAG_KEEP_POLICY | \ SCHED_FLAG_KEEP_PARAMS) @@ -143,6 +144,7 @@ struct clone_args { SCHED_FLAG_RECLAIM | \ SCHED_FLAG_DL_OVERRUN | \ SCHED_FLAG_KEEP_ALL | \ - SCHED_FLAG_UTIL_CLAMP) + SCHED_FLAG_UTIL_CLAMP | \ + SCHED_FLAG_LATENCY_NICE) =20 #endif /* _UAPI_LINUX_SCHED_H */ --=20 2.17.1 From nobody Tue Jun 23 03:14:49 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B59BC433F5 for ; Fri, 11 Mar 2022 16:14:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350216AbiCKQPo (ORCPT ); Fri, 11 Mar 2022 11:15:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39738 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350189AbiCKQPi (ORCPT ); Fri, 11 Mar 2022 11:15:38 -0500 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 478271B50D7 for ; Fri, 11 Mar 2022 08:14:23 -0800 (PST) Received: by mail-wr1-x432.google.com with SMTP id r6so13245220wrr.2 for ; Fri, 11 Mar 2022 08:14:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=UsEhnvvFHgaQjzBQyJf4AjmX5slWha5zqrOlGHdJd7o=; b=nMW5qngXigefmP0YsG7knNofZZNEK38X697gX8WPLTVRVFSKbnTzPgGk7D46v5VeAC VAzaWLfXXq1QIoBMJkuDok9flIV+FC9DTPxYqqou4Xud7ufZdXRjnPM5hVVTogX7IGTz dtXoahuNiH+r/rMy04o9E+rnnFg2pdnnG7JftEknbQKHHVdm9oREOKEHdLt7JbqR86uO Q/plhpDTwDq3bruNJLpyaB+izF6nfKJaytvREOhQ3a8i8VO7LRvpxLmIEQXgPt+//XJ8 XCR7A9DUCqLU+AaXLB1pHSBCB0uBv7Lq9qiK6x3qFmoja6qY84TFxIFscuC5SnojPciZ w6wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=UsEhnvvFHgaQjzBQyJf4AjmX5slWha5zqrOlGHdJd7o=; b=U2HicaxuOutvdxOY9sBr9VjcRinjDvl2GGXucf8DAE+rgCNnvpE+DSLwYFu/Cj8WlC nUIGo9vbAE9juvvBXF8l77ovo/jOk9r3kAAgMh6qNfj+q5S+LvM09aonQSuS/QXKcMeF Nr2LRNvLdRg9Ik108MrXKBAR7zMUR8P75Mxn4PomfNDpL/Crt1STQHfE7pgvvi85J5ln kentsw7IvaLGhmGBMYeyTi/ajBMV9HKaKrnnufl/4BTqgv03Dz3cH/wKlelIB4t+6ftN lU5nhCXXA9vqbWfV7MzE8sZlF5gv4BlsqHVM58IkDGdjLR1umu4nn7rptjgT+/+C94lz 1V7g== X-Gm-Message-State: AOAM532Q8hQAu/aI0R5LLevm8aY+Tyt1LUKtI7nHwnD36LEXMVFjnn03 HnGG1bcq8MjNaiad2LFYwQ0T5g== X-Google-Smtp-Source: ABdhPJyfLeZtrGD8qr1pbQSQd+rDwHWzBZKe88bVygqK/Q9QfnhTS8JgIl2JfKORHOcuQ0FCLwIq6g== X-Received: by 2002:a5d:50c5:0:b0:1f0:2111:8f74 with SMTP id f5-20020a5d50c5000000b001f021118f74mr7797356wrt.211.1647015261595; Fri, 11 Mar 2022 08:14:21 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:f:6020:70d9:405c:a1e4:4d23]) by smtp.gmail.com with ESMTPSA id 4-20020a056000154400b00203812ca383sm6464137wry.78.2022.03.11.08.14.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Mar 2022 08:14:20 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qais.yousef@arm.com, chris.hyser@oracle.com, pkondeti@codeaurora.org, valentin.schneider@arm.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, dhaval.giani@oracle.com, qperret@google.com, tim.c.chen@linux.intel.com, Vincent Guittot Subject: [PATCH 4/6] sched/core: Add permission checks for setting the latency_nice value Date: Fri, 11 Mar 2022 17:14:04 +0100 Message-Id: <20220311161406.23497-5-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220311161406.23497-1-vincent.guittot@linaro.org> References: <20220311161406.23497-1-vincent.guittot@linaro.org> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Parth Shah Since the latency_nice uses the similar infrastructure as NICE, use the already existing CAP_SYS_NICE security checks for the latency_nice. This should return -EPERM for the non-root user when trying to set the task latency_nice value to any lower than the current value. Signed-off-by: Parth Shah [rebase] Signed-off-by: Vincent Guittot --- kernel/sched/core.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 3edba1a38ecb..8f8b102a75c4 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -7360,6 +7360,10 @@ static int __sched_setscheduler(struct task_struct *= p, return -EINVAL; if (attr->sched_latency_nice < MIN_LATENCY_NICE) return -EINVAL; + /* Use the same security checks as NICE */ + if (attr->sched_latency_nice < p->latency_nice && + !capable(CAP_SYS_NICE)) + return -EPERM; } =20 if (pi) --=20 2.17.1 From nobody Tue Jun 23 03:14:49 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B0EBC433EF for ; Fri, 11 Mar 2022 16:14:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350223AbiCKQP4 (ORCPT ); Fri, 11 Mar 2022 11:15:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39932 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350191AbiCKQPk (ORCPT ); Fri, 11 Mar 2022 11:15:40 -0500 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 083511D06F5 for ; Fri, 11 Mar 2022 08:14:26 -0800 (PST) Received: by mail-wm1-x32d.google.com with SMTP id f20-20020a05600c155400b00389d73ceb43so1222641wmg.4 for ; Fri, 11 Mar 2022 08:14:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=gGDDU2Ei4o4n5rRc9Fl7x+xD8gsBmPHZg9Xz6lcqQr4=; b=hD43wxXy6cLQaiXbqvSCujqNLqOLjVk+fi3fsycX+agewdbDj9KgK+s2Fo72/QDrb4 uG6emQs6lWjzfpq+WPeCgQR5wrEFHVDxyRC+NU3rk2ASu7SXYH3VA6kVBZ7n2wGAMhay AhDiLx2ghDZGS3/oBpfUDnuBqN0KIHwbRWRSnDVqfIUfQtcCjQFAhpzHXLwAjcQtC2JK vFHSH4KziT62Y4X5yeVgWg+aglb3l9zqPduDVnBySkUUJ+f87F+EYMTCHWbMmDsVXTYR SVVXHCqvItGtbnWIiV+CmrK5Dk6peHnI8Pz/TAwn/azygq0OSJVka7//v8HRLz3h5cGa 0xbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=gGDDU2Ei4o4n5rRc9Fl7x+xD8gsBmPHZg9Xz6lcqQr4=; b=Qu3oc2/pm0UoEt7fIGo97ieon8thIDx9JME9oWt8+q/vA8dFkypZyAoMWt8Xt98xbR oO184xyHwKWbWeW0mh6weGXefQ8ympryg5HmncXead3nMVQvgq8t9HVTkOqpuKYIf6ZB NO1ZFlSrvJJKnLEfmGBaKgFDVLSDDvEPKc1ML8dzeNMYMcQKeBptRQeXmGJ4QDnOFC7y G3dKojTBrgGj/7vE6+mkR1oGabSTgnxB+GoxdQm7CVOc5JMUR6vXTWMLf7SClQev6VWz mu1OiZbrPogdEkL6lerU/YO8rN4J7y8CNh8CEecw+lZyAqZDpx4wfyPwziZkVQesZa8o V81Q== X-Gm-Message-State: AOAM532JzkYTHARIsbnb8tm7MA04kX3X9YzNOVEcVObhyseEeqZLAKPr LTNx8KhB1hotwS1EG2Z6UNb6Tw== X-Google-Smtp-Source: ABdhPJxO+w9VkJ1FBjPMKZLThI/FF+KsuvbdSFy1CitXuc3sICEswshGp9+0cHx2nyHh3bIl2lpiyw== X-Received: by 2002:a7b:c042:0:b0:389:7336:158b with SMTP id u2-20020a7bc042000000b003897336158bmr8279837wmc.15.1647015264428; Fri, 11 Mar 2022 08:14:24 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:f:6020:70d9:405c:a1e4:4d23]) by smtp.gmail.com with ESMTPSA id 4-20020a056000154400b00203812ca383sm6464137wry.78.2022.03.11.08.14.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Mar 2022 08:14:22 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qais.yousef@arm.com, chris.hyser@oracle.com, pkondeti@codeaurora.org, valentin.schneider@arm.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, dhaval.giani@oracle.com, qperret@google.com, tim.c.chen@linux.intel.com, Vincent Guittot Subject: [RFC 5/6] sched/fair: Take into account latency nice at wakeup Date: Fri, 11 Mar 2022 17:14:05 +0100 Message-Id: <20220311161406.23497-6-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220311161406.23497-1-vincent.guittot@linaro.org> References: <20220311161406.23497-1-vincent.guittot@linaro.org> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Take into account the nice latency priority of a thread when deciding to preempt the current running thread. We don't want to provide more CPU bandwidth to a thread but reorder the scheduling to run latency sensitive task first whenever possible. As long as a thread didn't use its bandwidth, it will be able to preempt the current thread. At the opposite, a thread with a low latency priority will preempt current thread at wakeup only to keep fair CPU bandwidth sharing. Otherwise it will wait for the tick to get its sched slice. curr vruntime | sysctl_sched_wakeup_granularity <--> ----------------------------------|----|-----------------------|-----------= ---- | |<---------------------> | . sysctl_sched_latency | . default/current latency entity | . | . 1111111111111111111111111111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-= 1-1- se preempts curr at wakeup ------>|<- se doesn't preempt curr -------------= ---- | . | . | . low latency entity | . ---------------------->| % of sysctl_sched_latency | 1111111111111111111111111111111111111111111111111111111111|0000|-1-1-1-1-1-= 1-1- preempt ------------------------------------------------->|<- do not preemp= t -- | . | . | . high latency entity | . |<-----------------------| . | % of sysctl_sched_latency . 111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1= -1-1 preempt->|<- se doesn't preempt curr --------------------------------------= ---- Tests results of nice latency impact on heavy load like hackbench: hackbench -l (2560 / group) -g group group latency 0 latency 19 1 1.450(+/- 0.60%) 1.376(+/- 0.54%) + 5% 4 1.537(+/- 1.75%) 1.335(+/- 1.81%) +13% 8 1.414(+/- 1.91%) 1.348(+/- 1.88%) + 5% 16 1.423(+/- 1.65%) 1.374(+/- 0.58%) + 3% hackbench -p -l (2560 / group) -g group group 1 1.416(+/- 3.45%) 0.886(+/- 0.54%) +37% 4 1.634(+/- 7.14%) 0.888(+/- 5.40%) +45% 8 1.449(+/- 2.14%) 0.804(+/- 4.55%) +44% 16 0.917(+/- 4.12%) 0.777(+/- 1.41%) +15% By deacreasing the latency prio, we reduce the number of preemption at wakeup and help hackbench making progress. Test results of nice latency impact on short live load like cyclictest while competing with heavy load like hackbench: hackbench -l 10000 -g group & cyclictest --policy other -D 5 -q -n latency 0 latency -20 group min avg max min avg max 0 16 17 28 15 17 27 1 61 382 10603 63 89 4628 4 52 437 15455 54 98 16238 8 56 728 38499 61 125 28983 16 53 1215 52207 61 183 80751 group =3D 0 means that hackbench is not running. The avg is significantly improved with nice latency -20 especially with large number of groups but min and max remain quite similar. If we add the histogram parameters to get details of latency, we have : hackbench -l 10000 -g 16 & cyclictest --policy other -D 5 -q -n -H 20000 --histfile data.txt latency 0 latency -20 Min Latencies: 63 62 Avg Latencies: 1397 218 Max Latencies: 44926 42291 50% latencies: 100 98 75% latencies: 762 116 85% latencies: 1118 126 90% latencies: 1540 130 95% latencies: 5610 138 99% latencies: 13738 266 With percentile details, we see the benefit of nice latency -20 as 1% of the latencies stays above 266us whereas the default latency has got 25% are above 762us and 10% over the 1ms. Signed-off-by: Vincent Guittot --- include/linux/sched.h | 4 ++- init/init_task.c | 2 +- kernel/sched/core.c | 32 +++++++++++++++++++---- kernel/sched/debug.c | 2 +- kernel/sched/fair.c | 60 +++++++++++++++++++++++++++++++++++++++++-- kernel/sched/sched.h | 12 +++++++++ 6 files changed, 102 insertions(+), 10 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 2aa889a59054..9aeb157e819b 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -560,6 +560,8 @@ struct sched_entity { unsigned long runnable_weight; #endif =20 + int latency_weight; + #ifdef CONFIG_SMP /* * Per entity load average tracking. @@ -779,7 +781,7 @@ struct task_struct { int static_prio; int normal_prio; unsigned int rt_priority; - int latency_nice; + int latency_prio; =20 struct sched_entity se; struct sched_rt_entity rt; diff --git a/init/init_task.c b/init/init_task.c index 2afa249c253b..e98c71f24981 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -78,7 +78,7 @@ struct task_struct init_task .prio =3D MAX_PRIO - 20, .static_prio =3D MAX_PRIO - 20, .normal_prio =3D MAX_PRIO - 20, - .latency_nice =3D 0, + .latency_prio =3D NICE_WIDTH - 20, .policy =3D SCHED_NORMAL, .cpus_ptr =3D &init_task.cpus_mask, .user_cpus_ptr =3D NULL, diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 8f8b102a75c4..547b0da01efe 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -1241,6 +1241,11 @@ static void set_load_weight(struct task_struct *p, b= ool update_load) } } =20 +static void set_latency_weight(struct task_struct *p) +{ + p->se.latency_weight =3D sched_latency_to_weight[p->latency_prio]; +} + #ifdef CONFIG_UCLAMP_TASK /* * Serializes updates of utilization clamp values @@ -4394,7 +4399,7 @@ int sched_fork(unsigned long clone_flags, struct task= _struct *p) p->prio =3D current->normal_prio; =20 /* Propagate the parent's latency requirements to the child as well */ - p->latency_nice =3D current->latency_nice; + p->latency_prio =3D current->latency_prio; =20 uclamp_fork(p); =20 @@ -4412,7 +4417,7 @@ int sched_fork(unsigned long clone_flags, struct task= _struct *p) p->prio =3D p->normal_prio =3D p->static_prio; set_load_weight(p, false); =20 - p->latency_nice =3D DEFAULT_LATENCY_NICE; + p->latency_prio =3D NICE_TO_LATENCY(0); /* * We don't need the reset flag anymore after the fork. It has * fulfilled its duty: @@ -4420,6 +4425,9 @@ int sched_fork(unsigned long clone_flags, struct task= _struct *p) p->sched_reset_on_fork =3D 0; } =20 + /* Once latency_prio is set, update the latency weight */ + set_latency_weight(p); + if (dl_prio(p->prio)) return -EAGAIN; else if (rt_prio(p->prio)) @@ -7361,7 +7369,7 @@ static int __sched_setscheduler(struct task_struct *p, if (attr->sched_latency_nice < MIN_LATENCY_NICE) return -EINVAL; /* Use the same security checks as NICE */ - if (attr->sched_latency_nice < p->latency_nice && + if (attr->sched_latency_nice < LATENCY_TO_NICE(p->latency_prio) && !capable(CAP_SYS_NICE)) return -EPERM; } @@ -7401,7 +7409,7 @@ static int __sched_setscheduler(struct task_struct *p, if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP) goto change; if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE && - attr->sched_latency_nice !=3D p->latency_nice) + attr->sched_latency_nice !=3D LATENCY_TO_NICE(p->latency_prio)) goto change; =20 p->sched_reset_on_fork =3D reset_on_fork; @@ -7942,7 +7950,7 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sch= ed_attr __user *, uattr, get_params(p, &kattr); kattr.sched_flags &=3D SCHED_FLAG_ALL; =20 - kattr.sched_latency_nice =3D p->latency_nice; + kattr.sched_latency_nice =3D LATENCY_TO_NICE(p->latency_prio); =20 #ifdef CONFIG_UCLAMP_TASK /* @@ -10954,6 +10962,20 @@ const u32 sched_prio_to_wmult[40] =3D { /* 15 */ 119304647, 148102320, 186737708, 238609294, 286331153, }; =20 +/* + * latency weight for wakeup preemption + */ +const int sched_latency_to_weight[40] =3D { + /* -20 */ 1024, 973, 922, 870, 819, + /* -15 */ 768, 717, 666, 614, 563, + /* -10 */ 512, 461, 410, 358, 307, + /* -5 */ 256, 205, 154, 102, 51, + /* 0 */ 0, -51, -102, -154, -205, + /* 5 */ -256, -307, -358, -410, -461, + /* 10 */ -512, -563, -614, -666, -717, + /* 15 */ -768, -819, -870, -922, -973, +}; + void call_trace_sched_update_nr_running(struct rq *rq, int count) { trace_sched_update_nr_running_tp(rq, count); diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c index 5d76a8927888..253e52ec73fb 100644 --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -1043,7 +1043,7 @@ void proc_sched_show_task(struct task_struct *p, stru= ct pid_namespace *ns, #endif P(policy); P(prio); - P(latency_nice); + P(latency_prio); if (task_has_dl_policy(p)) { P(dl.runtime); P(dl.deadline); diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 5c4bfffe8c2c..506c482a0e48 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5555,6 +5555,35 @@ static int sched_idle_cpu(int cpu) } #endif =20 +static void set_next_buddy(struct sched_entity *se); + +static void check_preempt_from_idle(struct cfs_rq *cfs, struct sched_entit= y *se) +{ + struct sched_entity *next; + + if (se->latency_weight <=3D 0) + return; + + if (cfs->nr_running <=3D 1) + return; + /* + * When waking from idle, we don't need to check to preempt at wakeup + * the idle thread and don't set next buddy as a candidate for being + * picked in priority. + * In case of simultaneous wakeup from idle, the latency sensitive tasks + * lost opportunity to preempt non sensitive tasks which woke up + * simultaneously. + */ + + if (cfs->next) + next =3D cfs->next; + else + next =3D __pick_first_entity(cfs); + + if (next && wakeup_preempt_entity(next, se) =3D=3D 1) + set_next_buddy(se); +} + /* * The enqueue_task method is called before nr_running is * increased. Here we update the fair scheduling stats and @@ -5648,6 +5677,9 @@ enqueue_task_fair(struct rq *rq, struct task_struct *= p, int flags) if (!task_new) update_overutilized_status(rq); =20 + if (rq->curr =3D=3D rq->idle) + check_preempt_from_idle(cfs_rq_of(&p->se), &p->se); + enqueue_throttle: if (cfs_bandwidth_used()) { /* @@ -5669,8 +5701,6 @@ enqueue_task_fair(struct rq *rq, struct task_struct *= p, int flags) hrtick_update(rq); } =20 -static void set_next_buddy(struct sched_entity *se); - /* * The dequeue_task method is called before nr_running is * decreased. We remove the task from the rbtree and @@ -6970,6 +7000,27 @@ balance_fair(struct rq *rq, struct task_struct *prev= , struct rq_flags *rf) } #endif /* CONFIG_SMP */ =20 +static long wakeup_latency_gran(int latency_weight) +{ + long thresh =3D sysctl_sched_latency; + + if (!latency_weight) + return 0; + + if (sched_feat(GENTLE_FAIR_SLEEPERS)) + thresh >>=3D 1; + + /* + * Clamp the delta to stay in the scheduler period range + * [-sysctl_sched_latency:sysctl_sched_latency] + */ + latency_weight =3D clamp_t(long, latency_weight, + -1 * NICE_LATENCY_WEIGHT_MAX, + NICE_LATENCY_WEIGHT_MAX); + + return (thresh * latency_weight) >> NICE_LATENCY_SHIFT; +} + static unsigned long wakeup_gran(struct sched_entity *se) { unsigned long gran =3D sysctl_sched_wakeup_granularity; @@ -7008,6 +7059,10 @@ static int wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se) { s64 gran, vdiff =3D curr->vruntime - se->vruntime; + int latency_weight =3D se->latency_weight - curr->latency_weight; + + latency_weight =3D min(latency_weight, se->latency_weight); + vdiff +=3D wakeup_latency_gran(latency_weight); =20 if (vdiff <=3D 0) return -1; @@ -7117,6 +7172,7 @@ static void check_preempt_wakeup(struct rq *rq, struc= t task_struct *p, int wake_ return; =20 update_curr(cfs_rq_of(se)); + if (wakeup_preempt_entity(se, pse) =3D=3D 1) { /* * Bias pick_next to pick the sched entity that is diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 456ad2159eb1..dd92aa9c36f9 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -122,6 +122,17 @@ extern void call_trace_sched_update_nr_running(struct = rq *rq, int count); * Default tasks should be treated as a task with latency_nice =3D 0. */ #define DEFAULT_LATENCY_NICE 0 +#define DEFAULT_LATENCY_PRIO (DEFAULT_LATENCY_NICE + LATENCY_NICE_WIDTH/2) + +/* + * Convert user-nice values [ -20 ... 0 ... 19 ] + * to static latency [ 0..39 ], + * and back. + */ +#define NICE_TO_LATENCY(nice) ((nice) + DEFAULT_LATENCY_PRIO) +#define LATENCY_TO_NICE(prio) ((prio) - DEFAULT_LATENCY_PRIO) +#define NICE_LATENCY_SHIFT (SCHED_FIXEDPOINT_SHIFT) +#define NICE_LATENCY_WEIGHT_MAX (1L << NICE_LATENCY_SHIFT) =20 /* * Increase resolution of nice-level calculations for 64-bit architectures. @@ -2098,6 +2109,7 @@ static_assert(WF_TTWU =3D=3D SD_BALANCE_WAKE); =20 extern const int sched_prio_to_weight[40]; extern const u32 sched_prio_to_wmult[40]; +extern const int sched_latency_to_weight[40]; =20 /* * {de,en}queue flags: --=20 2.17.1 From nobody Tue Jun 23 03:14:49 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C9E9C433F5 for ; Fri, 11 Mar 2022 16:14:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350189AbiCKQPt (ORCPT ); Fri, 11 Mar 2022 11:15:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39726 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350202AbiCKQPk (ORCPT ); Fri, 11 Mar 2022 11:15:40 -0500 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12DE11D0374 for ; Fri, 11 Mar 2022 08:14:28 -0800 (PST) Received: by mail-wr1-x42e.google.com with SMTP id t11so13651649wrm.5 for ; Fri, 11 Mar 2022 08:14:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=7rhgsqV3KMfKxCS9qLfTFgcLJX3t+C452Rt6S9wGKCk=; b=dHnTAZY8lk+sp8PiDY+JePLPNcH5273gsKX+kvojcT9GtJloUQBtNRkBxMCqiNXMZA c5QQeLbO4kImju6nmq90MTZaRKMTwE1WfCXJhlaLaMNd5jH2Ek1LYXkQYLDgZxPv1pmH FZCrRiGQ0XZ+HGyB+yZrOr00ufpgnYjCsxZQeVw4JJre28hVMEVEdGi5kKy/Te3XpmuO DmfGMMfkijgfogN7m2AarkVnv4I/WBOxidgrGGb7lc+Hyyby9wNQpr2OQKgswm5kPBBd 1x5+GupHlijDYZx7mZFdWqaimDG356ig73s2kZD76uxaMALTBV1Ca8hidvTg/DCIhRgZ cXHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=7rhgsqV3KMfKxCS9qLfTFgcLJX3t+C452Rt6S9wGKCk=; b=zSCoiZQjT/ExSo2HyAkqmKagCUW+9H9PgkZZ3ioZBZLvecj7RE9ZjIyQuO0eolcKVV wsNbSkXOUxx8j0063szcsYhSqyc0Bx2EzHHTweHeJhoV23MroHRYnGxD/UWo8lKDK5i+ nvfItixylqULypU/bh0FNAcrf0BRbUesRhbLvyjZLiW1uvVD5vrBf/RET47D3WWbVrML zbmrlG0DZmigbeyihhDoGfZFI56Gbe+wXh2kdkK4QayWVn0T+Yqwu05f5v2tGmkstLgA fnzTvRw1n6MVja4vU77WfwWySI+/cr1x4lBAyqz8R9RWPn15U2qKvMMDJ+hv6+VrXwi2 +p2Q== X-Gm-Message-State: AOAM533nJDgnnHgOX33cDP1YladAdxRpyscZrj4LhDPjrAwke70NxtZG Ie+dle/OJAv/n5g03E6mMsASBQ== X-Google-Smtp-Source: ABdhPJzPsNJ2oenReqzj3KbB6ZJjT/f7ZvPQkRptHu7xaCEh+CTyRWyh6r6Tt00oxY17cc9POYIyrw== X-Received: by 2002:adf:fb47:0:b0:1ed:9f2c:492e with SMTP id c7-20020adffb47000000b001ed9f2c492emr8013720wrs.196.1647015266504; Fri, 11 Mar 2022 08:14:26 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:f:6020:70d9:405c:a1e4:4d23]) by smtp.gmail.com with ESMTPSA id 4-20020a056000154400b00203812ca383sm6464137wry.78.2022.03.11.08.14.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Mar 2022 08:14:25 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qais.yousef@arm.com, chris.hyser@oracle.com, pkondeti@codeaurora.org, valentin.schneider@arm.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, dhaval.giani@oracle.com, qperret@google.com, tim.c.chen@linux.intel.com, Vincent Guittot Subject: [RFC 6/6] sched/fair: Add sched group latency support Date: Fri, 11 Mar 2022 17:14:06 +0100 Message-Id: <20220311161406.23497-7-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220311161406.23497-1-vincent.guittot@linaro.org> References: <20220311161406.23497-1-vincent.guittot@linaro.org> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Tasks can set its latency priority which is then used to decide to preempt the current running entity of the cfs but sched group entities still have the default latency priority. Add a latency field in task group to set the latency priority of the group which will be used against other entity in the parent cfs. Signed-off-by: Vincent Guittot --- kernel/sched/core.c | 41 +++++++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 32 ++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 4 ++++ 3 files changed, 77 insertions(+) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 547b0da01efe..e0668652dd24 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -10635,6 +10635,30 @@ static int cpu_idle_write_s64(struct cgroup_subsys= _state *css, { return sched_group_set_idle(css_tg(css), idle); } + +static s64 cpu_latency_read_s64(struct cgroup_subsys_state *css, + struct cftype *cft) +{ + return css_tg(css)->latency_prio; +} + +static int cpu_latency_write_s64(struct cgroup_subsys_state *css, + struct cftype *cft, s64 latency_prio) +{ + return sched_group_set_latency(css_tg(css), latency_prio); +} + +static s64 cpu_latency_nice_read_s64(struct cgroup_subsys_state *css, + struct cftype *cft) +{ + return LATENCY_TO_NICE(css_tg(css)->latency_prio); +} + +static int cpu_latency_nice_write_s64(struct cgroup_subsys_state *css, + struct cftype *cft, s64 latency_nice) +{ + return sched_group_set_latency(css_tg(css), NICE_TO_LATENCY(latency_nice)= ); +} #endif =20 static struct cftype cpu_legacy_files[] =3D { @@ -10649,6 +10673,11 @@ static struct cftype cpu_legacy_files[] =3D { .read_s64 =3D cpu_idle_read_s64, .write_s64 =3D cpu_idle_write_s64, }, + { + .name =3D "latency", + .read_s64 =3D cpu_latency_read_s64, + .write_s64 =3D cpu_latency_write_s64, + }, #endif #ifdef CONFIG_CFS_BANDWIDTH { @@ -10866,6 +10895,18 @@ static struct cftype cpu_files[] =3D { .read_s64 =3D cpu_idle_read_s64, .write_s64 =3D cpu_idle_write_s64, }, + { + .name =3D "latency", + .flags =3D CFTYPE_NOT_ON_ROOT, + .read_s64 =3D cpu_latency_read_s64, + .write_s64 =3D cpu_latency_write_s64, + }, + { + .name =3D "latency.nice", + .flags =3D CFTYPE_NOT_ON_ROOT, + .read_s64 =3D cpu_latency_nice_read_s64, + .write_s64 =3D cpu_latency_nice_write_s64, + }, #endif #ifdef CONFIG_CFS_BANDWIDTH { diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 506c482a0e48..cbccef025089 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -11496,6 +11496,7 @@ int alloc_fair_sched_group(struct task_group *tg, s= truct task_group *parent) goto err; =20 tg->shares =3D NICE_0_LOAD; + tg->latency_prio =3D DEFAULT_LATENCY_PRIO; =20 init_cfs_bandwidth(tg_cfs_bandwidth(tg)); =20 @@ -11594,6 +11595,7 @@ void init_tg_cfs_entry(struct task_group *tg, struc= t cfs_rq *cfs_rq, } =20 se->my_q =3D cfs_rq; + se->latency_weight =3D sched_latency_to_weight[tg->latency_prio]; /* guarantee group entities always have weight */ update_load_set(&se->load, NICE_0_LOAD); se->parent =3D parent; @@ -11724,6 +11726,36 @@ int sched_group_set_idle(struct task_group *tg, lo= ng idle) return 0; } =20 +int sched_group_set_latency(struct task_group *tg, long latency_prio) +{ + int i; + + if (tg =3D=3D &root_task_group) + return -EINVAL; + + if (latency_prio < 0 || + latency_prio > LATENCY_NICE_WIDTH) + return -EINVAL; + + mutex_lock(&shares_mutex); + + if (tg->latency_prio =3D=3D latency_prio) { + mutex_unlock(&shares_mutex); + return 0; + } + + tg->latency_prio =3D latency_prio; + + for_each_possible_cpu(i) { + struct sched_entity *se =3D tg->se[i]; + + WRITE_ONCE(se->latency_weight, sched_latency_to_weight[latency_prio]); + } + + mutex_unlock(&shares_mutex); + return 0; +} + #else /* CONFIG_FAIR_GROUP_SCHED */ =20 void free_fair_sched_group(struct task_group *tg) { } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index dd92aa9c36f9..885d1c809329 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -429,6 +429,8 @@ struct task_group { =20 /* A positive value indicates that this is a SCHED_IDLE group. */ int idle; + /* latency priority of the group. */ + int latency_prio; =20 #ifdef CONFIG_SMP /* @@ -542,6 +544,8 @@ extern int sched_group_set_shares(struct task_group *tg= , unsigned long shares); =20 extern int sched_group_set_idle(struct task_group *tg, long idle); =20 +extern int sched_group_set_latency(struct task_group *tg, long latency); + #ifdef CONFIG_SMP extern void set_task_rq_fair(struct sched_entity *se, struct cfs_rq *prev, struct cfs_rq *next); --=20 2.17.1