From nobody Thu Apr 2 11:43:29 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C989C54EE9 for ; Sun, 25 Sep 2022 14:41:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229548AbiIYOlV (ORCPT ); Sun, 25 Sep 2022 10:41:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40912 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230325AbiIYOlR (ORCPT ); Sun, 25 Sep 2022 10:41:17 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31CDE1105 for ; Sun, 25 Sep 2022 07:41:16 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id d12-20020a05600c3acc00b003b4c12e47f3so2392968wms.4 for ; Sun, 25 Sep 2022 07:41:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date; bh=oSSdXMqnb9jEQAlol2YdyD+HZ/O1tyTS1XWv0Aq1bWg=; b=cPHEcE8rvpBq+wrpZUPeVwFolO1d4AqBpn5RK/4Q2kFuaLwKE4flQC0JZ3Sx1VYUwn VYyg8Ymx9i5eC1JTiKu2fBkqUhaKBC1tE81t15VmqMOqndvj6SjNpBiDN7/rtTZwAfSr L/g9f2jOm8HXkKgEQWLCuP+Jx1mDr1zdmiYykDm3WHSvrU1DTudDEP60FXLWqqdh0fzk n+B0Gc7TTeZcxy5IEwfrnZ2+VtvqjV3x68Bwjyau1NNd+4OOAF0ip52hq2hemdsIjl4T u+TljLKLnf8ScuEXRq1+iWixRLSKZU4wlURs2ZfRdkIw+ofduqOHRlf1ix7Jx0YGgtGS SzEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date; bh=oSSdXMqnb9jEQAlol2YdyD+HZ/O1tyTS1XWv0Aq1bWg=; b=Bv3KHrFQVIIFVyUCpv0mVvvywinQWsYBLQLN0Ea7o6qJxOLRPjKpmQ/dEA4oQoR+1t D9xKBD0sVMZgWF8DBAFRyKyUZ4rHP9EaRYj6BTXtd1GUEwy2JMQMOmZhjdTO4q3nzpWQ To0mMK0Qy9D5oMEEw6bwm6uteJ9Sr8So/uzCIeyQhvH/S6PzdmGB6SfqYxwp921Uvj/H RsOoro4mmZyM5lCTCdq0j58iyUz24B4cQ58CDmJdtkRGgfDpbmMD5MVR+cJNPuIJe/b9 Azg/cYYjZNtzYf/WkR9yH8aGI6gkx025yPb3PrCoDo0ICxTgkJmj0JgrgrlBwMIvNPpH vgkA== X-Gm-Message-State: ACrzQf3MajZihhH+iHjjuQFHjXuCKx5MbGqfasQG6mKe2NzXXwYAF4fq ZT3rwxHDKRSq0jHumk2ByGSgCg== X-Google-Smtp-Source: AMsMyM6nl1MuYVtLHYyjYzrrUXNiLpdSNYU6mn3qAP8bNIBYG5B1PRxaBBiyiQluWfURdtdLz4IOLg== X-Received: by 2002:a7b:cbc4:0:b0:3b4:fd67:d70d with SMTP id n4-20020a7bcbc4000000b003b4fd67d70dmr13819004wmi.98.1664116874616; Sun, 25 Sep 2022 07:41:14 -0700 (PDT) Received: from localhost.localdomain (91-160-61-128.subs.proxad.net. [91.160.61.128]) by smtp.gmail.com with ESMTPSA id m10-20020a05600c3b0a00b003b47b913901sm20774761wms.1.2022.09.25.07.41.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Sep 2022 07:41:13 -0700 (PDT) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qais.yousef@arm.com, chris.hyser@oracle.com, valentin.schneider@arm.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, Vincent Guittot Subject: [PATCH v5 1/7] sched: Introduce latency-nice as a per-task attribute Date: Sun, 25 Sep 2022 16:39:02 +0200 Message-Id: <20220925143908.10846-2-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220925143908.10846-1-vincent.guittot@linaro.org> References: <20220925143908.10846-1-vincent.guittot@linaro.org> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Parth Shah Latency-nice indicates the latency requirements of a task with respect to the other tasks in the system. The value of the attribute can be within the range of [-20, 19] both inclusive to be in-line with the values just like task nice values. latency_nice =3D -20 indicates the task to have the least latency as compared to the tasks having latency_nice =3D +19. The latency_nice may affect only the CFS SCHED_CLASS by getting latency requirements from the userspace. Additionally, add debugging bits for newly added latency_nice attribute. Signed-off-by: Parth Shah [rebase] Signed-off-by: Vincent Guittot --- include/linux/sched.h | 1 + kernel/sched/debug.c | 1 + kernel/sched/sched.h | 18 ++++++++++++++++++ 3 files changed, 20 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index 15e3bd96e4ce..6805f378a9c3 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -783,6 +783,7 @@ struct task_struct { int static_prio; int normal_prio; unsigned int rt_priority; + int latency_nice; =20 struct sched_entity se; struct sched_rt_entity rt; diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c index bb3d63bdf4ae..a3f7876217a6 100644 --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -1042,6 +1042,7 @@ void proc_sched_show_task(struct task_struct *p, stru= ct pid_namespace *ns, #endif P(policy); P(prio); + P(latency_nice); if (task_has_dl_policy(p)) { P(dl.runtime); P(dl.deadline); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 1fc198be1ffd..eeb6efb0b610 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -125,6 +125,24 @@ extern int sched_rr_timeslice; */ #define NS_TO_JIFFIES(TIME) ((unsigned long)(TIME) / (NSEC_PER_SEC / HZ)) =20 +/* + * Latency nice is meant to provide scheduler hints about the relative + * latency requirements of a task with respect to other tasks. + * Thus a task with latency_nice =3D=3D 19 can be hinted as the task with = no + * latency requirements, in contrast to the task with latency_nice =3D=3D = -20 + * which should be given priority in terms of lower latency. + */ +#define MAX_LATENCY_NICE 19 +#define MIN_LATENCY_NICE -20 + +#define LATENCY_NICE_WIDTH \ + (MAX_LATENCY_NICE - MIN_LATENCY_NICE + 1) + +/* + * Default tasks should be treated as a task with latency_nice =3D 0. + */ +#define DEFAULT_LATENCY_NICE 0 + /* * Increase resolution of nice-level calculations for 64-bit architectures. * The extra resolution improves shares distribution and load balancing of --=20 2.17.1 From nobody Thu Apr 2 11:43:29 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F9F1C54EE9 for ; Sun, 25 Sep 2022 14:41:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231506AbiIYOl0 (ORCPT ); Sun, 25 Sep 2022 10:41:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40952 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230379AbiIYOlS (ORCPT ); Sun, 25 Sep 2022 10:41:18 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 037C11105 for ; Sun, 25 Sep 2022 07:41:17 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id y5so6730927wrh.3 for ; Sun, 25 Sep 2022 07:41:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date; bh=xgZILBJrfBdVYZE/N+pW4Q56k/FtOZrt0ohTpwnzbzE=; b=tTH9OXxH0XyodBLjOoQeDn7GlN70ICdhQTxyV2KVUMfvfjTsbGsI4+9PPGUgRsydV6 Z7vLG+Io4LPU5mmFDXT3bCXi2cQKnMoRRQSWizVp9SXYEZEycGIbX412wIhB/RSUO7pF L4zaWZnODe4fxGon8+kdP0aCm0UYqwKG+F3fmcllFsK+S8j0hh+AhIbv709lEHZDU75E +ov4eXCJAy+8SrGLeWNp58OxAxOqzrbaytjV4yKwPpXlddq2V5LSbYhahaeCOdIWXNJG GGoblvYz9rGYTV/+B77X7Qzt5ZRxvZ/OzbyigtF7ACQ261N9qzIEIgivI4GYTDxwO3nM 89nA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date; bh=xgZILBJrfBdVYZE/N+pW4Q56k/FtOZrt0ohTpwnzbzE=; b=vDnue+LZgCRelSEY5M4pPFkFq/uJVjpAQQWLgkFw3S3hXOdc/KbSOCKYMoAOmo1yO3 j3s+KeX7UGQUMc9cSCUr/beLgHwm7LVg90Kluy4hMBVYgUFHUrb2ZYNMXBHOO0lum+Qy W0ORG3i71PQ7NW6WVvDeYS9fG9alLy1tAjxySs6W+JEqVIebVsAeEKsqRalddVKt+O8Z 7xrlE0UlOzsO/Utk387eo05YWK9TrceunUaBB+ZKDCP7NBcR9iB2Z83L2woGndej9pgW VhrBxemF7W5ysGsPJmCZrG29H7rSqVrtjPyRdQp3xnA/V3gD22CP23BOzw/ehaDiTefZ JZkA== X-Gm-Message-State: ACrzQf1w0K7PIYN6JcoX6aiQgBdr+vYi910yaywBAjVB2VqUBED48Odl zwc1Qmnm19QxERmzaUOuRpTgBXQuLJxBaA== X-Google-Smtp-Source: AMsMyM5Rje3rn7B9bY/c6pQAH2N0ln9osv11EiK4TqfUTb3HKIqQXa2lQMFBafIfqzugO23tmH0xfg== X-Received: by 2002:a5d:6c62:0:b0:22a:2f59:cb7d with SMTP id r2-20020a5d6c62000000b0022a2f59cb7dmr10791596wrz.405.1664116876293; Sun, 25 Sep 2022 07:41:16 -0700 (PDT) Received: from localhost.localdomain (91-160-61-128.subs.proxad.net. [91.160.61.128]) by smtp.gmail.com with ESMTPSA id m10-20020a05600c3b0a00b003b47b913901sm20774761wms.1.2022.09.25.07.41.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Sep 2022 07:41:15 -0700 (PDT) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qais.yousef@arm.com, chris.hyser@oracle.com, valentin.schneider@arm.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, Vincent Guittot Subject: [PATCH v5 2/7] sched/core: Propagate parent task's latency requirements to the child task Date: Sun, 25 Sep 2022 16:39:03 +0200 Message-Id: <20220925143908.10846-3-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220925143908.10846-1-vincent.guittot@linaro.org> References: <20220925143908.10846-1-vincent.guittot@linaro.org> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Parth Shah Clone parent task's latency_nice attribute to the forked child task. Reset the latency_nice value to default value when the child task is set to sched_reset_on_fork. Also, initialize init_task.latency_nice value with DEFAULT_LATENCY_NICE value Signed-off-by: Parth Shah [rebase] Signed-off-by: Vincent Guittot --- init/init_task.c | 1 + kernel/sched/core.c | 1 + 2 files changed, 2 insertions(+) diff --git a/init/init_task.c b/init/init_task.c index ff6c4b9bfe6b..7dd71dd2d261 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -78,6 +78,7 @@ struct task_struct init_task .prio =3D MAX_PRIO - 20, .static_prio =3D MAX_PRIO - 20, .normal_prio =3D MAX_PRIO - 20, + .latency_nice =3D DEFAULT_LATENCY_NICE, .policy =3D SCHED_NORMAL, .cpus_ptr =3D &init_task.cpus_mask, .user_cpus_ptr =3D NULL, diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 4fa4a3ddb4f4..ada2d05bd894 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4559,6 +4559,7 @@ int sched_fork(unsigned long clone_flags, struct task= _struct *p) p->prio =3D p->normal_prio =3D p->static_prio; set_load_weight(p, false); =20 + p->latency_nice =3D DEFAULT_LATENCY_NICE; /* * We don't need the reset flag anymore after the fork. It has * fulfilled its duty: --=20 2.17.1 From nobody Thu Apr 2 11:43:29 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6F61C54EE9 for ; Sun, 25 Sep 2022 14:41:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230303AbiIYOlb (ORCPT ); Sun, 25 Sep 2022 10:41:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41006 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230519AbiIYOlU (ORCPT ); Sun, 25 Sep 2022 10:41:20 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 635912DA88 for ; Sun, 25 Sep 2022 07:41:19 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id cc5so6708295wrb.6 for ; Sun, 25 Sep 2022 07:41:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date; bh=e8Biwda7XCN9clNdZx9AGbx8CtyTGBWBnyh3Zf/Yxd0=; b=jgL9Mkfl15VFp5U5ziDU4SKe+GH8lm4BxCYaSH/myS/Q+YLbdDYD8PwowR7Ca6KNAH +9eCKTWT31S3uSkDDlNpkniMHlKVg6p6zpA3T9OCt5LDol2V/SEl+xrkkqdg4JKvS9CH /aBLUiOma8XNEdEoIJIeKqkoT2WWeJt3wJqiy/NuD3EDzW6Spn8TVCJkObtwnrz0/TdC jfQ/V3PaeK+NHBOGguRorCJ8f+Z91IdRuzaQZkCwFrWW5oMJCpkOUr2LRYfCJJjolWst iyM7UruEiq7tPhliHTRi/D3UBOXdGk7T+PyfeNARWWmcLVZUC0sV50h0vPd4QELk3MHC DZgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date; bh=e8Biwda7XCN9clNdZx9AGbx8CtyTGBWBnyh3Zf/Yxd0=; b=vhlWg0XzhbBGZ6Tkwt8GYhxA2jtt0MRN4vrsHzo8nTgIbUNAgQJDonlCXcbNjlfUoT uDGEwGPENeMx/XNZ//we2mtoWdys8/2ojmxjB0+DN7W5gbGZTgcB4PxVunSiG/AhMQaY HW9sxLe9xwgDk/6Z1yLEZTeX2tMSFAppPwUQ2C3AnzQ3ATt05OFwhCay/76DLwkXgvzz 9rnyk9vBDVDj4gLkMjFN8rPOwPSFddMVrZgkJGsSYMG8HGgd5zZq0tWuCUef/t+EyBa3 jl80JWtcf3/HbocrtUlqH5jRd2EET1g+LWoQtEUwAbKYCikwFyiwvnCZpmRdhtJ/kzTU T8IA== X-Gm-Message-State: ACrzQf3JGxQpfXmOWgqzOBjQnvm/+Gj1r9OwHEoTegnx7zSZj2bi955y eu/7jdBzffDlHkL+cL4ynN1W6G+/bK28lg== X-Google-Smtp-Source: AMsMyM4wepcG+kYWNcqe5JSkBKbR6iJJ255Xy1WVZwBNH0dl3G5sGu8fHbxGR2KHct2EZbbfl4bKpg== X-Received: by 2002:a5d:4887:0:b0:226:ed34:7bbd with SMTP id g7-20020a5d4887000000b00226ed347bbdmr10394940wrq.561.1664116877834; Sun, 25 Sep 2022 07:41:17 -0700 (PDT) Received: from localhost.localdomain (91-160-61-128.subs.proxad.net. [91.160.61.128]) by smtp.gmail.com with ESMTPSA id m10-20020a05600c3b0a00b003b47b913901sm20774761wms.1.2022.09.25.07.41.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Sep 2022 07:41:17 -0700 (PDT) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qais.yousef@arm.com, chris.hyser@oracle.com, valentin.schneider@arm.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, Vincent Guittot Subject: [PATCH v5 3/7] sched: Allow sched_{get,set}attr to change latency_nice of the task Date: Sun, 25 Sep 2022 16:39:04 +0200 Message-Id: <20220925143908.10846-4-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220925143908.10846-1-vincent.guittot@linaro.org> References: <20220925143908.10846-1-vincent.guittot@linaro.org> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Parth Shah Introduce the latency_nice attribute to sched_attr and provide a mechanism to change the value with the use of sched_setattr/sched_getattr syscall. Also add new flag "SCHED_FLAG_LATENCY_NICE" to hint the change in latency_nice of the task on every sched_setattr syscall. Signed-off-by: Parth Shah [rebase and add a dedicated __setscheduler_latency ] Signed-off-by: Vincent Guittot --- include/uapi/linux/sched.h | 4 +++- include/uapi/linux/sched/types.h | 19 +++++++++++++++++++ kernel/sched/core.c | 24 ++++++++++++++++++++++++ tools/include/uapi/linux/sched.h | 4 +++- 4 files changed, 49 insertions(+), 2 deletions(-) diff --git a/include/uapi/linux/sched.h b/include/uapi/linux/sched.h index 3bac0a8ceab2..b2e932c25be6 100644 --- a/include/uapi/linux/sched.h +++ b/include/uapi/linux/sched.h @@ -132,6 +132,7 @@ struct clone_args { #define SCHED_FLAG_KEEP_PARAMS 0x10 #define SCHED_FLAG_UTIL_CLAMP_MIN 0x20 #define SCHED_FLAG_UTIL_CLAMP_MAX 0x40 +#define SCHED_FLAG_LATENCY_NICE 0x80 =20 #define SCHED_FLAG_KEEP_ALL (SCHED_FLAG_KEEP_POLICY | \ SCHED_FLAG_KEEP_PARAMS) @@ -143,6 +144,7 @@ struct clone_args { SCHED_FLAG_RECLAIM | \ SCHED_FLAG_DL_OVERRUN | \ SCHED_FLAG_KEEP_ALL | \ - SCHED_FLAG_UTIL_CLAMP) + SCHED_FLAG_UTIL_CLAMP | \ + SCHED_FLAG_LATENCY_NICE) =20 #endif /* _UAPI_LINUX_SCHED_H */ diff --git a/include/uapi/linux/sched/types.h b/include/uapi/linux/sched/ty= pes.h index f2c4589d4dbf..db1e8199e8c8 100644 --- a/include/uapi/linux/sched/types.h +++ b/include/uapi/linux/sched/types.h @@ -10,6 +10,7 @@ struct sched_param { =20 #define SCHED_ATTR_SIZE_VER0 48 /* sizeof first published struct */ #define SCHED_ATTR_SIZE_VER1 56 /* add: util_{min,max} */ +#define SCHED_ATTR_SIZE_VER2 60 /* add: latency_nice */ =20 /* * Extended scheduling parameters data structure. @@ -98,6 +99,22 @@ struct sched_param { * scheduled on a CPU with no more capacity than the specified value. * * A task utilization boundary can be reset by setting the attribute to -1. + * + * Latency Tolerance Attributes + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D + * + * A subset of sched_attr attributes allows to specify the relative latency + * requirements of a task with respect to the other tasks running/queued i= n the + * system. + * + * @ sched_latency_nice task's latency_nice value + * + * The latency_nice of a task can have any value in a range of + * [MIN_LATENCY_NICE..MAX_LATENCY_NICE]. + * + * A task with latency_nice with the value of LATENCY_NICE_MIN can be + * taken for a task requiring a lower latency as opposed to the task with + * higher latency_nice. */ struct sched_attr { __u32 size; @@ -120,6 +137,8 @@ struct sched_attr { __u32 sched_util_min; __u32 sched_util_max; =20 + /* latency requirement hints */ + __s32 sched_latency_nice; }; =20 #endif /* _UAPI_LINUX_SCHED_TYPES_H */ diff --git a/kernel/sched/core.c b/kernel/sched/core.c index ada2d05bd894..6a6116ea4c2c 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -7318,6 +7318,14 @@ static void __setscheduler_params(struct task_struct= *p, p->rt_priority =3D attr->sched_priority; p->normal_prio =3D normal_prio(p); set_load_weight(p, true); + +} + +static void __setscheduler_latency(struct task_struct *p, + const struct sched_attr *attr) +{ + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) + p->latency_nice =3D attr->sched_latency_nice; } =20 /* @@ -7460,6 +7468,13 @@ static int __sched_setscheduler(struct task_struct *= p, return retval; } =20 + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) { + if (attr->sched_latency_nice > MAX_LATENCY_NICE) + return -EINVAL; + if (attr->sched_latency_nice < MIN_LATENCY_NICE) + return -EINVAL; + } + if (pi) cpuset_read_lock(); =20 @@ -7494,6 +7509,9 @@ static int __sched_setscheduler(struct task_struct *p, goto change; if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP) goto change; + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE && + attr->sched_latency_nice !=3D p->latency_nice) + goto change; =20 p->sched_reset_on_fork =3D reset_on_fork; retval =3D 0; @@ -7582,6 +7600,7 @@ static int __sched_setscheduler(struct task_struct *p, __setscheduler_params(p, attr); __setscheduler_prio(p, newprio); } + __setscheduler_latency(p, attr); __setscheduler_uclamp(p, attr); =20 if (queued) { @@ -7792,6 +7811,9 @@ static int sched_copy_attr(struct sched_attr __user *= uattr, struct sched_attr *a size < SCHED_ATTR_SIZE_VER1) return -EINVAL; =20 + if ((attr->sched_flags & SCHED_FLAG_LATENCY_NICE) && + size < SCHED_ATTR_SIZE_VER2) + return -EINVAL; /* * XXX: Do we want to be lenient like existing syscalls; or do we want * to be strict and return an error on out-of-bounds values? @@ -8029,6 +8051,8 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sch= ed_attr __user *, uattr, get_params(p, &kattr); kattr.sched_flags &=3D SCHED_FLAG_ALL; =20 + kattr.sched_latency_nice =3D p->latency_nice; + #ifdef CONFIG_UCLAMP_TASK /* * This could race with another potential updater, but this is fine diff --git a/tools/include/uapi/linux/sched.h b/tools/include/uapi/linux/sc= hed.h index 3bac0a8ceab2..ecc4884bfe4b 100644 --- a/tools/include/uapi/linux/sched.h +++ b/tools/include/uapi/linux/sched.h @@ -132,6 +132,7 @@ struct clone_args { #define SCHED_FLAG_KEEP_PARAMS 0x10 #define SCHED_FLAG_UTIL_CLAMP_MIN 0x20 #define SCHED_FLAG_UTIL_CLAMP_MAX 0x40 +#define SCHED_FLAG_LATENCY_NICE 0X80 =20 #define SCHED_FLAG_KEEP_ALL (SCHED_FLAG_KEEP_POLICY | \ SCHED_FLAG_KEEP_PARAMS) @@ -143,6 +144,7 @@ struct clone_args { SCHED_FLAG_RECLAIM | \ SCHED_FLAG_DL_OVERRUN | \ SCHED_FLAG_KEEP_ALL | \ - SCHED_FLAG_UTIL_CLAMP) + SCHED_FLAG_UTIL_CLAMP | \ + SCHED_FLAG_LATENCY_NICE) =20 #endif /* _UAPI_LINUX_SCHED_H */ --=20 2.17.1 From nobody Thu Apr 2 11:43:29 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8EA88C54EE9 for ; Sun, 25 Sep 2022 14:41:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232228AbiIYOlo (ORCPT ); Sun, 25 Sep 2022 10:41:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41116 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231390AbiIYOlX (ORCPT ); Sun, 25 Sep 2022 10:41:23 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A51A82DA8F for ; Sun, 25 Sep 2022 07:41:21 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id n15so6711271wrq.5 for ; Sun, 25 Sep 2022 07:41:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date; bh=McxzoSyGgN0mMu4LmDTxJ3mXTH5u5yE8fp9dw11YuUg=; b=Mw07vpbbd2RUDvXabW7GB3HZUgWS+JdnKszKXl5YvMnuIjIeoqL70Tj3xqjwFXLOPJ OmR/x/ow7c7DLgIQ1L1Ipx6TEP3ecbOlhNwGovIacDFzlOYydOkFsBasHIx7tNimketO 50hr6awGxlc54VdC4Vbti/XB84tl5riBKZbz2D0RhBQfQjndbagYjByYoOXH7pdDOStO B4A8kDZaGHIuUzAJ6ZmJ/GBgahOv6NBSLUoWfbs3noM2OohZdtbnm+Ndq+HPxK127+5O xBRK1RRsveJVwoeZoy8hQSWP62XlTp0nM+GWd+Q++J50r5eiWk+YEP3eVpmNdS1Et+1x gPhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date; bh=McxzoSyGgN0mMu4LmDTxJ3mXTH5u5yE8fp9dw11YuUg=; b=W7LGex4PavGpIjPXe0WD0GFxfmb0qJcorW+J8pD7ceqsveFmxN0p7yVypsWIUk9eAA 6sjnrjNmwNW1hfah6XsyycyslnU7Lz8Re4Una+nrkEty6MqAu3OZkhzWDAXSgZm54Pu/ As50Fr8GTGqTKGg4vkGx0Kw9XWoWfib1F11bn+isTjjy8RqaQe0GEQ3moFlvvo4kqyff HCOxLvTtmTipwEj3GZKJjykSMRoIrjhBFVe4b2W4IeKOOa0U71T/qCoY7OHnjnQp4h0h A6tv65er2Ak81RWB7I9PvDu7p7fiNbXn+pmKSMTsBBtpuDnfeSadrZqF2BuTfBLaEg2b SLcQ== X-Gm-Message-State: ACrzQf3eqHbH93sa1EWqTau/htQx7B+mzwijmxrpRtdGlpsd7SD6BT/J OBJugDCTtcclLBCeLDHWFdQCbw== X-Google-Smtp-Source: AMsMyM6IoHp/09oB6vJRkG1m6Z+zzow5eOg6xIJwcQsE0547aDBHNXzfNHA5b3QF+2Soy02ANmA4/A== X-Received: by 2002:a05:6000:1565:b0:22c:8da7:3cf8 with SMTP id 5-20020a056000156500b0022c8da73cf8mr5643302wrz.688.1664116879959; Sun, 25 Sep 2022 07:41:19 -0700 (PDT) Received: from localhost.localdomain (91-160-61-128.subs.proxad.net. [91.160.61.128]) by smtp.gmail.com with ESMTPSA id m10-20020a05600c3b0a00b003b47b913901sm20774761wms.1.2022.09.25.07.41.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Sep 2022 07:41:18 -0700 (PDT) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qais.yousef@arm.com, chris.hyser@oracle.com, valentin.schneider@arm.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, Vincent Guittot Subject: [PATCH v5 4/7] sched/fair: Take into account latency priority at wakeup Date: Sun, 25 Sep 2022 16:39:05 +0200 Message-Id: <20220925143908.10846-5-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220925143908.10846-1-vincent.guittot@linaro.org> References: <20220925143908.10846-1-vincent.guittot@linaro.org> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Take into account the latency priority of a thread when deciding to preempt the current running thread. We don't want to provide more CPU bandwidth to a thread but reorder the scheduling to run latency sensitive task first whenever possible. As long as a thread didn't use its bandwidth, it will be able to preempt the current thread. At the opposite, a thread with a low latency priority will preempt current thread at wakeup only to keep fair CPU bandwidth sharing. Otherwise it will wait for the tick to get its sched slice. curr vruntime | sysctl_sched_wakeup_granularity <--> ----------------------------------|----|-----------------------|-----------= ---- | |<---------------------> | . sysctl_sched_latency | . default/current latency entity | . | . 1111111111111111111111111111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-= 1-1- se preempts curr at wakeup ------>|<- se doesn't preempt curr -------------= ---- | . | . | . low latency entity | . ---------------------->| % of sysctl_sched_latency | 1111111111111111111111111111111111111111111111111111111111|0000|-1-1-1-1-1-= 1-1- preempt ------------------------------------------------->|<- do not preemp= t -- | . | . | . high latency entity | . |<-----------------------| . | % of sysctl_sched_latency . 111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1= -1-1 preempt->|<- se doesn't preempt curr --------------------------------------= ---- Tests results of nice latency impact on heavy load like hackbench: hackbench -l (2560 / group) -g group group latency 0 latency 19 1 1.350(+/- 1.10%) 1.287(+/- 1.60%) + 5% 4 1.392(+/- 1.67%) 1.248(+/- 1.51%) +11% 8 1.294(+/- 1.56%) 1.254(+/- 1.96%) + 3% 16 1.315(+/- 1.02%) 1.288(+/- 1.53%) + 2% hackbench -p -l (2560 / group) -g group group 1 1.768(+/- 13%) 0.805(+/- 2%) +54% 4 1.634(+/- 13%) 0.765(+/- 1%) +53% 8 1.305(+/- 4%) 0.745(+/- 2%) +43% 16 0.786(+/- 4%) 0.705(+/- 2%) +10% By deacreasing the latency prio, we reduce the number of preemption at wakeup and help hackbench making progress. Test results of nice latency impact on short live load like cyclictest while competing with heavy load like hackbench: hackbench -l 10000 -g $group & cyclictest --policy other -D 5 -q -n latency 0 latency -20 group min avg max min avg max 0 17 18 28 16 18 28 1 67 319 7369 63 76 2479 4 64 453 51699 45 95 8788 8 65 814 51699 63 116 19535 16 64 1275 37744 63 159 51134 group =3D 0 means that hackbench is not running. The avg is significantly improved with nice latency -20 especially with large number of groups but min and max remain quite similar. If we add the histogram parameter to get details of latency, we have : hackbench -l 10000 -g 16 & cyclictest --policy other -D 5 -q -n -H 20000 --histfile data.txt latency 0 latency -20 Min Latencies: 63 63 Avg Latencies: 1459 214 Max Latencies: 58274 82358 50% latencies: 164 87 75% latencies: 848 91 85% latencies: 1246 94 90% latencies: 2149 96 95% latencies: 8463 99 99% latencies:>20000 120 With percentile details, we see the benefit of nice latency -20 as only 1% of the latencies stays above 120us whereas the default latency has got 25% are above 848us, 10% over the 2ms and 1% above 20ms. Signed-off-by: Vincent Guittot --- include/linux/sched.h | 4 +++- init/init_task.c | 2 +- kernel/sched/core.c | 38 +++++++++++++++++++++++++---- kernel/sched/debug.c | 2 +- kernel/sched/fair.c | 56 +++++++++++++++++++++++++++++++++++++++++-- kernel/sched/sched.h | 14 ++++++++++- 6 files changed, 105 insertions(+), 11 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 6805f378a9c3..a74cad08e91e 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -567,6 +567,8 @@ struct sched_entity { /* cached value of my_q->h_nr_running */ unsigned long runnable_weight; #endif + /* preemption offset in ns */ + long latency_offset; =20 #ifdef CONFIG_SMP /* @@ -783,7 +785,7 @@ struct task_struct { int static_prio; int normal_prio; unsigned int rt_priority; - int latency_nice; + int latency_prio; =20 struct sched_entity se; struct sched_rt_entity rt; diff --git a/init/init_task.c b/init/init_task.c index 7dd71dd2d261..b8ddf403bc62 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -78,7 +78,7 @@ struct task_struct init_task .prio =3D MAX_PRIO - 20, .static_prio =3D MAX_PRIO - 20, .normal_prio =3D MAX_PRIO - 20, - .latency_nice =3D DEFAULT_LATENCY_NICE, + .latency_prio =3D NICE_WIDTH - 20, .policy =3D SCHED_NORMAL, .cpus_ptr =3D &init_task.cpus_mask, .user_cpus_ptr =3D NULL, diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 6a6116ea4c2c..00fa2da12506 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -1284,6 +1284,16 @@ static void set_load_weight(struct task_struct *p, b= ool update_load) } } =20 +static void set_latency_offset(struct task_struct *p) +{ + long weight =3D sched_latency_to_weight[p->latency_prio]; + s64 offset; + + offset =3D sysctl_sched_latency * weight; + offset =3D div_s64(offset, NICE_LATENCY_WEIGHT_MAX); + p->se.latency_offset =3D (long)offset; +} + #ifdef CONFIG_UCLAMP_TASK /* * Serializes updates of utilization clamp values @@ -4559,7 +4569,9 @@ int sched_fork(unsigned long clone_flags, struct task= _struct *p) p->prio =3D p->normal_prio =3D p->static_prio; set_load_weight(p, false); =20 - p->latency_nice =3D DEFAULT_LATENCY_NICE; + p->latency_prio =3D NICE_TO_LATENCY(0); + set_latency_offset(p); + /* * We don't need the reset flag anymore after the fork. It has * fulfilled its duty: @@ -7324,8 +7336,10 @@ static void __setscheduler_params(struct task_struct= *p, static void __setscheduler_latency(struct task_struct *p, const struct sched_attr *attr) { - if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) - p->latency_nice =3D attr->sched_latency_nice; + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) { + p->latency_prio =3D NICE_TO_LATENCY(attr->sched_latency_nice); + set_latency_offset(p); + } } =20 /* @@ -7510,7 +7524,7 @@ static int __sched_setscheduler(struct task_struct *p, if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP) goto change; if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE && - attr->sched_latency_nice !=3D p->latency_nice) + attr->sched_latency_nice !=3D LATENCY_TO_NICE(p->latency_prio)) goto change; =20 p->sched_reset_on_fork =3D reset_on_fork; @@ -8051,7 +8065,7 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sch= ed_attr __user *, uattr, get_params(p, &kattr); kattr.sched_flags &=3D SCHED_FLAG_ALL; =20 - kattr.sched_latency_nice =3D p->latency_nice; + kattr.sched_latency_nice =3D LATENCY_TO_NICE(p->latency_prio); =20 #ifdef CONFIG_UCLAMP_TASK /* @@ -11204,6 +11218,20 @@ const u32 sched_prio_to_wmult[40] =3D { /* 15 */ 119304647, 148102320, 186737708, 238609294, 286331153, }; =20 +/* + * latency weight for wakeup preemption + */ +const int sched_latency_to_weight[40] =3D { + /* -20 */ -1024, -973, -922, -870, -819, + /* -15 */ -768, -717, -666, -614, -563, + /* -10 */ -512, -461, -410, -358, -307, + /* -5 */ -256, -205, -154, -102, -51, + /* 0 */ 0, 51, 102, 154, 205, + /* 5 */ 256, 307, 358, 410, 461, + /* 10 */ 512, 563, 614, 666, 717, + /* 15 */ 768, 819, 870, 922, 973, +}; + void call_trace_sched_update_nr_running(struct rq *rq, int count) { trace_sched_update_nr_running_tp(rq, count); diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c index a3f7876217a6..06aaa0c81d4b 100644 --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -1042,7 +1042,7 @@ void proc_sched_show_task(struct task_struct *p, stru= ct pid_namespace *ns, #endif P(policy); P(prio); - P(latency_nice); + P(latency_prio); if (task_has_dl_policy(p)) { P(dl.runtime); P(dl.deadline); diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 5ffec4370602..c3f857630dcf 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4574,6 +4574,8 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_en= tity *se, int flags) update_idle_cfs_rq_clock_pelt(cfs_rq); } =20 +static long wakeup_latency_gran(struct sched_entity *curr, struct sched_en= tity *se); + /* * Preempt the current task with a newly woken task if needed: */ @@ -4606,6 +4608,7 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sche= d_entity *curr) =20 se =3D __pick_first_entity(cfs_rq); delta =3D curr->vruntime - se->vruntime; + delta -=3D wakeup_latency_gran(curr, se); =20 if (delta < 0) return; @@ -5732,6 +5735,35 @@ static int sched_idle_cpu(int cpu) } #endif =20 +static void set_next_buddy(struct sched_entity *se); + +static void check_preempt_from_others(struct cfs_rq *cfs, struct sched_ent= ity *se) +{ + struct sched_entity *next; + + if (se->latency_offset >=3D 0) + return; + + if (cfs->nr_running <=3D 1) + return; + /* + * When waking from another class, we don't need to check to preempt at + * wakeup and don't set next buddy as a candidate for being picked in + * priority. + * In case of simultaneous wakeup when current is another class, the + * latency sensitive tasks lost opportunity to preempt non sensitive + * tasks which woke up simultaneously. + */ + + if (cfs->next) + next =3D cfs->next; + else + next =3D __pick_first_entity(cfs); + + if (next && wakeup_preempt_entity(next, se) =3D=3D 1) + set_next_buddy(se); +} + /* * The enqueue_task method is called before nr_running is * increased. Here we update the fair scheduling stats and @@ -5818,14 +5850,15 @@ enqueue_task_fair(struct rq *rq, struct task_struct= *p, int flags) if (!task_new) update_overutilized_status(rq); =20 + if (rq->curr->sched_class !=3D &fair_sched_class) + check_preempt_from_others(cfs_rq_of(&p->se), &p->se); + enqueue_throttle: assert_list_leaf_cfs_rq(rq); =20 hrtick_update(rq); } =20 -static void set_next_buddy(struct sched_entity *se); - /* * The dequeue_task method is called before nr_running is * decreased. We remove the task from the rbtree and @@ -7144,6 +7177,22 @@ balance_fair(struct rq *rq, struct task_struct *prev= , struct rq_flags *rf) } #endif /* CONFIG_SMP */ =20 +static long wakeup_latency_gran(struct sched_entity *curr, struct sched_en= tity *se) +{ + long latency_offset =3D se->latency_offset; + + /* + * A negative latency offset means that the sched_entity has latency + * requirement that needs to be evaluated versus other entity. + * Otherwise, use the latency weight to evaluate how much scheduling + * delay is acceptable by se. + */ + if ((se->latency_offset < 0) || (curr->latency_offset < 0)) + latency_offset -=3D curr->latency_offset; + + return latency_offset; +} + static unsigned long wakeup_gran(struct sched_entity *se) { unsigned long gran =3D sysctl_sched_wakeup_granularity; @@ -7183,6 +7232,9 @@ wakeup_preempt_entity(struct sched_entity *curr, stru= ct sched_entity *se) { s64 gran, vdiff =3D curr->vruntime - se->vruntime; =20 + /* Take into account latency priority */ + vdiff -=3D wakeup_latency_gran(curr, se); + if (vdiff <=3D 0) return -1; =20 diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index eeb6efb0b610..80c4d2f5827f 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -142,6 +142,17 @@ extern int sched_rr_timeslice; * Default tasks should be treated as a task with latency_nice =3D 0. */ #define DEFAULT_LATENCY_NICE 0 +#define DEFAULT_LATENCY_PRIO (DEFAULT_LATENCY_NICE + LATENCY_NICE_WIDTH/2) + +/* + * Convert user-nice values [ -20 ... 0 ... 19 ] + * to static latency [ 0..39 ], + * and back. + */ +#define NICE_TO_LATENCY(nice) ((nice) + DEFAULT_LATENCY_PRIO) +#define LATENCY_TO_NICE(prio) ((prio) - DEFAULT_LATENCY_PRIO) +#define NICE_LATENCY_SHIFT (SCHED_FIXEDPOINT_SHIFT) +#define NICE_LATENCY_WEIGHT_MAX (1L << NICE_LATENCY_SHIFT) =20 /* * Increase resolution of nice-level calculations for 64-bit architectures. @@ -2116,6 +2127,7 @@ static_assert(WF_TTWU =3D=3D SD_BALANCE_WAKE); =20 extern const int sched_prio_to_weight[40]; extern const u32 sched_prio_to_wmult[40]; +extern const int sched_latency_to_weight[40]; =20 /* * {de,en}queue flags: @@ -2450,8 +2462,8 @@ extern void check_preempt_curr(struct rq *rq, struct = task_struct *p, int flags); extern const_debug unsigned int sysctl_sched_nr_migrate; extern const_debug unsigned int sysctl_sched_migration_cost; =20 -#ifdef CONFIG_SCHED_DEBUG extern unsigned int sysctl_sched_latency; +#ifdef CONFIG_SCHED_DEBUG extern unsigned int sysctl_sched_min_granularity; extern unsigned int sysctl_sched_idle_min_granularity; extern unsigned int sysctl_sched_wakeup_granularity; --=20 2.17.1 From nobody Thu Apr 2 11:43:29 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA118C54EE9 for ; Sun, 25 Sep 2022 14:41:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230219AbiIYOls (ORCPT ); Sun, 25 Sep 2022 10:41:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41476 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231176AbiIYOlg (ORCPT ); Sun, 25 Sep 2022 10:41:36 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 838622DABB for ; Sun, 25 Sep 2022 07:41:23 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id t7so6667861wrm.10 for ; Sun, 25 Sep 2022 07:41:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date; bh=o98Owo/cazdQCIpyZbsHknX+Bzko5QxNVdsJZAzn3+o=; b=JEfeEN3NoI1t9hP1zimNRatHAKYgiG2LcewfCuDPWKnmAa2d98GEb4u6ir+INOOq9S GSnPDj38YK0xAOMl6ARNoRJdSR0m8V6yKPNbEkjhIhzUNB5eebT70D4wQlnU4Jl24+Ml u/jKl6GU0ayXQ7YdGMLq+O1RoTTi/7g3OmSUHlddldFd7finCpJ16qLKhopu94PR4qKo nLz0t8ZYIOgtDkiFUlTC8bRMWWSSVRmDMGxCyHeIuOYmSQKrQ8k9s45CRVB/jUXKdQYc mQqpJtkQZJ5PPnTgRQsxhnBGexquCtMDU3RBm1fKzgSU+qPkaHNYd8VAydAQKkac4OvE keUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date; bh=o98Owo/cazdQCIpyZbsHknX+Bzko5QxNVdsJZAzn3+o=; b=7X1bc9pO+ms6jfpDZNV80tWd8DGzNE9EHfmw0NTqdgXcmHh9tVJEocooUnHezLXXYM Fc1+oOQFkLOmlOQgW+9QynZx9vzprsZZPQbLo+MlSeZ8Pl9o8ckbOp8Pg17334CvMoUL KSHfBNDy5qPG5MkUMqnZj/RskLeVBEgI6UGjOZYk+lDUHg7bI2b/d55DQfU7biDhDxCA 1z3NeQhserFxKNjN1b81vCOY2e3kjySwZjBZTWXL4gsgXPZLvL+e64EpmCItl8i9lVTu eNFQdL60PF2eV+QYeYKZZYdWxZ04mW9lCCLBIkqd5hZxGRZDJG3DEb0a4P/vZImW7qZd djWg== X-Gm-Message-State: ACrzQf29CCoN1KyPEOFNFBjpYv6PhSb20oxK1g2+s0b9koX/cUFd9AG8 9zo+NHhBbU8UqT9UCKOCuYagGA== X-Google-Smtp-Source: AMsMyM7Ncanlah9exUhnT80/ZanFQ7pPEX1lTu2dpg7wexA4/x1UQwdOajD3SQIVPMZ/oWliErGrFg== X-Received: by 2002:a5d:6b49:0:b0:225:6e92:22f5 with SMTP id x9-20020a5d6b49000000b002256e9222f5mr10336647wrw.529.1664116881981; Sun, 25 Sep 2022 07:41:21 -0700 (PDT) Received: from localhost.localdomain (91-160-61-128.subs.proxad.net. [91.160.61.128]) by smtp.gmail.com with ESMTPSA id m10-20020a05600c3b0a00b003b47b913901sm20774761wms.1.2022.09.25.07.41.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Sep 2022 07:41:20 -0700 (PDT) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qais.yousef@arm.com, chris.hyser@oracle.com, valentin.schneider@arm.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, Vincent Guittot Subject: [PATCH v5 5/7] sched/fair: Add sched group latency support Date: Sun, 25 Sep 2022 16:39:06 +0200 Message-Id: <20220925143908.10846-6-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220925143908.10846-1-vincent.guittot@linaro.org> References: <20220925143908.10846-1-vincent.guittot@linaro.org> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Task can set its latency priority with sched_setattr(), which is then used to set the latency offset of its sched_entity, but sched group entities still have the default latency offset value. Add a latency.nice field in cpu cgroup controller to set the latency priority of the group similarly to sched_setattr(). The latency priority is then used to set the offset of the sched_entities of the group. Signed-off-by: Vincent Guittot --- Documentation/admin-guide/cgroup-v2.rst | 8 ++++ kernel/sched/core.c | 53 +++++++++++++++++++++++++ kernel/sched/fair.c | 33 +++++++++++++++ kernel/sched/sched.h | 4 ++ 4 files changed, 98 insertions(+) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-= guide/cgroup-v2.rst index be4a77baf784..d8ae7e411f9c 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -1095,6 +1095,14 @@ All time durations are in microseconds. values similar to the sched_setattr(2). This maximum utilization value is used to clamp the task specific maximum utilization clamp. =20 + cpu.latency.nice + A read-write single value file which exists on non-root + cgroups. The default is "0". + + The nice value is in the range [-20, 19]. + + This interface file allows reading and setting latency using the + same values used by sched_setattr(2). =20 =20 Memory diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 00fa2da12506..e8a1105bc87d 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -10890,6 +10890,48 @@ static int cpu_idle_write_s64(struct cgroup_subsys= _state *css, { return sched_group_set_idle(css_tg(css), idle); } + +static s64 cpu_latency_nice_read_s64(struct cgroup_subsys_state *css, + struct cftype *cft) +{ + int last_delta =3D INT_MAX; + int prio, delta; + s64 weight; + + weight =3D css_tg(css)->latency_offset * NICE_LATENCY_WEIGHT_MAX; + weight =3D div_s64(weight, sysctl_sched_latency); + + /* Find the closest nice value to the current weight */ + for (prio =3D 0; prio < ARRAY_SIZE(sched_latency_to_weight); prio++) { + delta =3D abs(sched_latency_to_weight[prio] - weight); + if (delta >=3D last_delta) + break; + last_delta =3D delta; + } + + return LATENCY_TO_NICE(prio-1); +} + +static int cpu_latency_nice_write_s64(struct cgroup_subsys_state *css, + struct cftype *cft, s64 nice) +{ + s64 latency_offset; + long weight; + int idx; + + if (nice < MIN_LATENCY_NICE || nice > MAX_LATENCY_NICE) + return -ERANGE; + + idx =3D NICE_TO_LATENCY(nice); + idx =3D array_index_nospec(idx, LATENCY_NICE_WIDTH); + weight =3D sched_latency_to_weight[idx]; + + latency_offset =3D sysctl_sched_latency * weight; + latency_offset =3D div_s64(latency_offset, NICE_LATENCY_WEIGHT_MAX); + + return sched_group_set_latency(css_tg(css), latency_offset); +} + #endif =20 static struct cftype cpu_legacy_files[] =3D { @@ -10904,6 +10946,11 @@ static struct cftype cpu_legacy_files[] =3D { .read_s64 =3D cpu_idle_read_s64, .write_s64 =3D cpu_idle_write_s64, }, + { + .name =3D "latency.nice", + .read_s64 =3D cpu_latency_nice_read_s64, + .write_s64 =3D cpu_latency_nice_write_s64, + }, #endif #ifdef CONFIG_CFS_BANDWIDTH { @@ -11121,6 +11168,12 @@ static struct cftype cpu_files[] =3D { .read_s64 =3D cpu_idle_read_s64, .write_s64 =3D cpu_idle_write_s64, }, + { + .name =3D "latency.nice", + .flags =3D CFTYPE_NOT_ON_ROOT, + .read_s64 =3D cpu_latency_nice_read_s64, + .write_s64 =3D cpu_latency_nice_write_s64, + }, #endif #ifdef CONFIG_CFS_BANDWIDTH { diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index c3f857630dcf..74e42d19c1ce 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -11768,6 +11768,7 @@ int alloc_fair_sched_group(struct task_group *tg, s= truct task_group *parent) goto err; =20 tg->shares =3D NICE_0_LOAD; + tg->latency_offset =3D 0; =20 init_cfs_bandwidth(tg_cfs_bandwidth(tg)); =20 @@ -11866,6 +11867,9 @@ void init_tg_cfs_entry(struct task_group *tg, struc= t cfs_rq *cfs_rq, } =20 se->my_q =3D cfs_rq; + + se->latency_offset =3D tg->latency_offset; + /* guarantee group entities always have weight */ update_load_set(&se->load, NICE_0_LOAD); se->parent =3D parent; @@ -11996,6 +12000,35 @@ int sched_group_set_idle(struct task_group *tg, lo= ng idle) return 0; } =20 +int sched_group_set_latency(struct task_group *tg, s64 latency) +{ + int i; + + if (tg =3D=3D &root_task_group) + return -EINVAL; + + if (abs(latency) > sysctl_sched_latency) + return -EINVAL; + + mutex_lock(&shares_mutex); + + if (tg->latency_offset =3D=3D latency) { + mutex_unlock(&shares_mutex); + return 0; + } + + tg->latency_offset =3D latency; + + for_each_possible_cpu(i) { + struct sched_entity *se =3D tg->se[i]; + + WRITE_ONCE(se->latency_offset, latency); + } + + mutex_unlock(&shares_mutex); + return 0; +} + #else /* CONFIG_FAIR_GROUP_SCHED */ =20 void free_fair_sched_group(struct task_group *tg) { } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 80c4d2f5827f..a15fb955092c 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -407,6 +407,8 @@ struct task_group { =20 /* A positive value indicates that this is a SCHED_IDLE group. */ int idle; + /* latency constraint of the group. */ + int latency_offset; =20 #ifdef CONFIG_SMP /* @@ -517,6 +519,8 @@ extern int sched_group_set_shares(struct task_group *tg= , unsigned long shares); =20 extern int sched_group_set_idle(struct task_group *tg, long idle); =20 +extern int sched_group_set_latency(struct task_group *tg, s64 latency); + #ifdef CONFIG_SMP extern void set_task_rq_fair(struct sched_entity *se, struct cfs_rq *prev, struct cfs_rq *next); --=20 2.17.1 From nobody Thu Apr 2 11:43:29 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21DE5C54EE9 for ; Sun, 25 Sep 2022 14:41:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232535AbiIYOly (ORCPT ); Sun, 25 Sep 2022 10:41:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41466 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231916AbiIYOlm (ORCPT ); Sun, 25 Sep 2022 10:41:42 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8E7B92E6A0 for ; Sun, 25 Sep 2022 07:41:25 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id c11so6659190wrp.11 for ; Sun, 25 Sep 2022 07:41:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date; bh=g+vXRBBqadahqyZybKZY8U1Su9Fe7IsjjckgHlRaHyY=; b=ZLms5hFb/jMOK439ZrX1sSwiGCETd0lgRNblA6iGtcnhGlcZXSLxoN7/e9x01Ht/Ol ubu0LKwgcCxNayU3FR4KgljEf577uw6hiWWs7FtxG2Bdfe0qM0BAyc/2HbeKp3ypz/z7 4EpfYOJQlVpY/jy6P+hrllcvU5z37HpmStG08mII6xDZHjQrwqyz1IftJBwSeSxlNBYK PdUOpISuqlW8dbgjgdAXAG/1RYGmcW3kysVmr+BL2/OeYbn/N4q2xnXvGRMbFz6teuRf yfhRXr0uu0LjlEV/4By+th+ejrv1WmqJH7lq4Br7vc2ZGjfE8IvHjY6fiyhi24Dkv6hY YUkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date; bh=g+vXRBBqadahqyZybKZY8U1Su9Fe7IsjjckgHlRaHyY=; b=jcvmFNnwKos7FRCKLJnkpFVc2/JjqjpdxdPsmMRTTKkthX9nckdK5EiwlMA91SkIqA aEZ+r5/31+5vtT12ioDstDm96qC8VqvC/jgrnE05I0v9sBbwCHufZ9PkE5Er5VgxYCjA IS7Qt12ir+LatTFeQ9cKSJ5TDAZe2xdeTIavteTAnB4SMzkoyW0kQM67jIJw0oZJnF4a Hy9R3gK0wTvX8O0C//bn9Sv2mSpdQdFcuMZqQZZogJM1JymU53XUFpQzNRk+njkEnpLq NewUgXndGVv+PDQbHIGmFMrJGpEgrPpdAg+inweBvsdCa/dSZ3hJFCj7Rog4XPO3qB7o /1LQ== X-Gm-Message-State: ACrzQf3eL7OzmN9yHpw4g4yefnifTO1yuX8XZVs42jPpQxbE/XoEW7u+ MdWpuWAkylkl8OgnxJGLzI8fZw== X-Google-Smtp-Source: AMsMyM7pIOq3LeafdXwIqfk3WII8jVRxxYMLI+qQ4zZdyobRrthJvmSbgMllsIFIP/aN4lhmDalMYw== X-Received: by 2002:a05:6000:1f8e:b0:22a:ff55:ea48 with SMTP id bw14-20020a0560001f8e00b0022aff55ea48mr10242604wrb.121.1664116883592; Sun, 25 Sep 2022 07:41:23 -0700 (PDT) Received: from localhost.localdomain (91-160-61-128.subs.proxad.net. [91.160.61.128]) by smtp.gmail.com with ESMTPSA id m10-20020a05600c3b0a00b003b47b913901sm20774761wms.1.2022.09.25.07.41.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Sep 2022 07:41:22 -0700 (PDT) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qais.yousef@arm.com, chris.hyser@oracle.com, valentin.schneider@arm.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, Vincent Guittot Subject: [PATCH v5 6/7] sched/core: Support latency priority with sched core Date: Sun, 25 Sep 2022 16:39:07 +0200 Message-Id: <20220925143908.10846-7-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220925143908.10846-1-vincent.guittot@linaro.org> References: <20220925143908.10846-1-vincent.guittot@linaro.org> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Take into account wakeup_latency_gran() when ordering the cfs threads. Signed-off-by: Vincent Guittot --- kernel/sched/fair.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 74e42d19c1ce..e524e892d118 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -11443,6 +11443,10 @@ bool cfs_prio_less(struct task_struct *a, struct t= ask_struct *b, bool in_fi) delta =3D (s64)(sea->vruntime - seb->vruntime) + (s64)(cfs_rqb->min_vruntime_fi - cfs_rqa->min_vruntime_fi); =20 + /* Take into account latency prio */ + delta -=3D wakeup_latency_gran(sea, seb); + + return delta > 0; } #else --=20 2.17.1 From nobody Thu Apr 2 11:43:29 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 284D2C54EE9 for ; Sun, 25 Sep 2022 14:42:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229965AbiIYOl7 (ORCPT ); Sun, 25 Sep 2022 10:41:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41678 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232081AbiIYOlm (ORCPT ); Sun, 25 Sep 2022 10:41:42 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F06E62E6AF for ; Sun, 25 Sep 2022 07:41:25 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id cc5so6708597wrb.6 for ; Sun, 25 Sep 2022 07:41:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date; bh=BV3/18h/jr2lIISScNcNJ/91zKWUfVi7jy8Z+Z1DShU=; b=XbT1dB4mNJ4eB+Q7AFKXUYtJ33nk03jdgzqcstnPteWe5lo2FPzZpxIchq9zIOcxPt qH9PI7pYQQuhqJE89dhiOemceB28zXcMEMw4yN8ro2c2BSAJlT3IfpPgcJNF6SsEfkfD 3IP/JUBBDquY1olKwAJHj5v0IEz3YQLJncnQ2jd/B1pb1LI0tO786QjTqg7RIDv1Rzpq Eky63SA47+2O8OrqAzgjbwBuEzXDEksdsGcKUHfgmhu8Owfc374P/PEvoPR3zoLLcBzb vZImRgzBaMsPVYMsCGbyUZXy3+kLfoElhXI8q6M8xQlvS25Jz7Fuh+SxiyQ7600t+LlD fFfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date; bh=BV3/18h/jr2lIISScNcNJ/91zKWUfVi7jy8Z+Z1DShU=; b=o5YSWrlqfCYzQkVpyicOIcUVq6ol38C5ZAat90ASG3w6Pb/+aqQh2RvpSHFC73+E5y Iy0w0r4RwUDrXd9zs2ttVkk7tGnzVx7e41Uzd4M8//m7c2g1QWLLBRW/lnku2UJXrdwk Q0qcBfV0QVYM9u7ornXGV2FztmVAhPmhZBWgGgTsCVj2y9JYIDuJxGJXyExbxQnAKX11 XSQSU38BnShg6QLfEbx/NS9V69t+9dyNVLXe+2Z8OAghHlTZ6NOJultcPonXq1HzMy97 IQQYKQs2vrCHCoy8vxf8cWCIjnyE9j/51HOmQ0X1kJScHAhQV0vAcsQPvdT2RjeDOPlb 2a5w== X-Gm-Message-State: ACrzQf2QwbXKATTo6eyYU9bRCV7r6+irVjsqjUanyQEq0rXWs7mdtV5k DOW+HzEDlI7KW2KR9S8itT4L4A== X-Google-Smtp-Source: AMsMyM4MIWmrjndc/K0E0+Gc4+sqsIH4/ppuy5AqmcnXCoDXxc8jnSVQ4W0q2qHbZh5+L7sLY9EtXA== X-Received: by 2002:a05:6000:188a:b0:22a:e4b7:c2f4 with SMTP id a10-20020a056000188a00b0022ae4b7c2f4mr10425776wri.446.1664116885091; Sun, 25 Sep 2022 07:41:25 -0700 (PDT) Received: from localhost.localdomain (91-160-61-128.subs.proxad.net. [91.160.61.128]) by smtp.gmail.com with ESMTPSA id m10-20020a05600c3b0a00b003b47b913901sm20774761wms.1.2022.09.25.07.41.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Sep 2022 07:41:24 -0700 (PDT) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qais.yousef@arm.com, chris.hyser@oracle.com, valentin.schneider@arm.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, Vincent Guittot Subject: [PATCH v5 7/7] sched/fair: Add latency list Date: Sun, 25 Sep 2022 16:39:08 +0200 Message-Id: <20220925143908.10846-8-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220925143908.10846-1-vincent.guittot@linaro.org> References: <20220925143908.10846-1-vincent.guittot@linaro.org> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Add a rb tree for latency sensitive entities so we can schedule the most sensitive one first even when it failed to preempt current at wakeup or when it got quickly preempted by another entity of higher priority. In order to keep fairness, the latency is used once at wakeup to get a minimum slice and not during the following scheduling slice to prevent long running entity to got more running time than allocated to his nice priority. The rb tree nebales to cover the last corner case where latency sensitive entity can't got schedule quickly after the wakeup. hackbench -l 10000 -g $group & cyclictest --policy other -D 5 -q -n latency 0 latency -20 group min avg max min avg max 0 17 19 29 17 18 30 1 65 306 7149 64 83 208 4 50 395 15731 56 80 271 8 56 781 41548 54 80 301 16 60 1392 87237 59 86 490 group =3D 0 means that hackbench is not running. Both avg and max are significantly improved with nice latency -20. If we add the histogram parameters to get details of latency, we have : hackbench -l 10000 -g 16 & cyclictest --policy other -D 5 -q -n -H 20000 --histfile data.txt latency 0 latency -20 Min Latencies: 60 61 Avg Latencies: 1077 86 Max Latencies: 87311 444 50% latencies: 92 85 75% latencies: 554 90 85% latencies: 1019 93 90% latencies: 1346 96 95% latencies: 5400 100 99% latencies: 19044 110 Signed-off-by: Vincent Guittot --- include/linux/sched.h | 2 + kernel/sched/fair.c | 96 +++++++++++++++++++++++++++++++++++++++++-- kernel/sched/sched.h | 1 + 3 files changed, 96 insertions(+), 3 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index a74cad08e91e..0b92674e3664 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -547,6 +547,8 @@ struct sched_entity { /* For load-balancing: */ struct load_weight load; struct rb_node run_node; + struct rb_node latency_node; + unsigned int on_latency; struct list_head group_node; unsigned int on_rq; =20 diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index e524e892d118..1a72f34136d8 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -664,7 +664,77 @@ struct sched_entity *__pick_last_entity(struct cfs_rq = *cfs_rq) =20 return __node_2_se(last); } +#endif =20 +/************************************************************** + * Scheduling class tree data structure manipulation methods: + * for latency + */ + +static inline bool latency_before(struct sched_entity *a, + struct sched_entity *b) +{ + return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_o= ffset) < 0; +} + +#define __latency_node_2_se(node) \ + rb_entry((node), struct sched_entity, latency_node) + +static inline bool __latency_less(struct rb_node *a, const struct rb_node = *b) +{ + return latency_before(__latency_node_2_se(a), __latency_node_2_se(b)); +} + +/* + * Enqueue an entity into the latency rb-tree: + */ +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *= se, int flags) +{ + + /* Only latency sensitive entity can be added to the list */ + if (se->latency_offset >=3D 0) + return; + + if (se->on_latency) + return; + + /* + * An execution time less than sysctl_sched_min_granularity means that + * the entity has been preempted by a higher sched class or an entity + * with higher latency constraint. + * Put it back in the list so it gets a chance to run 1st during the + * next slice. + */ + if (!(flags & ENQUEUE_WAKEUP)) { + u64 delta_exec =3D se->sum_exec_runtime - se->prev_sum_exec_runtime; + + if (delta_exec >=3D sysctl_sched_min_granularity) + return; + } + + rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_les= s); + se->on_latency =3D 1; +} + +static void __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *= se) +{ + if (se->on_latency) { + rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline); + se->on_latency =3D 0; + } +} + +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq) +{ + struct rb_node *left =3D rb_first_cached(&cfs_rq->latency_timeline); + + if (!left) + return NULL; + + return __latency_node_2_se(left); +} + +#ifdef CONFIG_SCHED_DEBUG /************************************************************** * Scheduling class statistics methods: */ @@ -4455,8 +4525,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_e= ntity *se, int flags) check_schedstat_required(); update_stats_enqueue_fair(cfs_rq, se, flags); check_spread(cfs_rq, se); - if (!curr) + if (!curr) { __enqueue_entity(cfs_rq, se); + __enqueue_latency(cfs_rq, se, flags); + } se->on_rq =3D 1; =20 if (cfs_rq->nr_running =3D=3D 1) { @@ -4542,8 +4614,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_e= ntity *se, int flags) =20 clear_buddies(cfs_rq, se); =20 - if (se !=3D cfs_rq->curr) + if (se !=3D cfs_rq->curr) { __dequeue_entity(cfs_rq, se); + __dequeue_latency(cfs_rq, se); + } se->on_rq =3D 0; account_entity_dequeue(cfs_rq, se); =20 @@ -4631,6 +4705,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_e= ntity *se) */ update_stats_wait_end_fair(cfs_rq, se); __dequeue_entity(cfs_rq, se); + __dequeue_latency(cfs_rq, se); update_load_avg(cfs_rq, se, UPDATE_TG); } =20 @@ -4669,7 +4744,7 @@ static struct sched_entity * pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr) { struct sched_entity *left =3D __pick_first_entity(cfs_rq); - struct sched_entity *se; + struct sched_entity *latency, *se; =20 /* * If curr is set we have to see if its left of the leftmost entity @@ -4711,6 +4786,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched= _entity *curr) se =3D cfs_rq->last; } =20 + /* Check for latency sensitive entity waiting for running */ + latency =3D __pick_first_latency(cfs_rq); + if (latency && (latency !=3D se) && + wakeup_preempt_entity(latency, se) < 1) + se =3D latency; + return se; } =20 @@ -4734,6 +4815,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, st= ruct sched_entity *prev) update_stats_wait_start_fair(cfs_rq, prev); /* Put 'current' back into the tree. */ __enqueue_entity(cfs_rq, prev); + __enqueue_latency(cfs_rq, prev, 0); /* in !on_rq case, update occurred at dequeue */ update_load_avg(cfs_rq, prev, 0); } @@ -11717,6 +11799,7 @@ static void set_next_task_fair(struct rq *rq, struc= t task_struct *p, bool first) void init_cfs_rq(struct cfs_rq *cfs_rq) { cfs_rq->tasks_timeline =3D RB_ROOT_CACHED; + cfs_rq->latency_timeline =3D RB_ROOT_CACHED; u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20))); #ifdef CONFIG_SMP raw_spin_lock_init(&cfs_rq->removed.lock); @@ -12025,8 +12108,15 @@ int sched_group_set_latency(struct task_group *tg,= s64 latency) =20 for_each_possible_cpu(i) { struct sched_entity *se =3D tg->se[i]; + struct rq *rq =3D cpu_rq(i); + struct rq_flags rf; + + rq_lock_irqsave(rq, &rf); =20 + __dequeue_latency(se->cfs_rq, se); WRITE_ONCE(se->latency_offset, latency); + + rq_unlock_irqrestore(rq, &rf); } =20 mutex_unlock(&shares_mutex); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index a15fb955092c..76bca172585c 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -599,6 +599,7 @@ struct cfs_rq { #endif =20 struct rb_root_cached tasks_timeline; + struct rb_root_cached latency_timeline; =20 /* * 'curr' points to currently running entity on this cfs_rq. --=20 2.17.1