From nobody Fri Oct 10 13:38:09 2025 Received: from mail-wr1-f54.google.com (mail-wr1-f54.google.com [209.85.221.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 771C22E6D12 for ; Fri, 13 Jun 2025 14:05:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749823532; cv=none; b=EuQcLa5wFXHKv4zNU3gD4HYt0TN64Iqn7zBL9QCafntFDm5l7j1QZVjgLuO7qfcmmXN+6O5Sh/QN17w+7tTp334ReqTIhC2nvgGnp9fm7DRnvkShyX1/k/0YMprOw1tfZq3cK/fpukjaJWxpOi/kIiM0/BMuLih4abOEdt8A8yE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749823532; c=relaxed/simple; bh=N3iswA1+2DOgy7vZ7wIfGsqALlE/MMqX5CZyUip3DqQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VgeKI4FKr0ZLrZfLePqETYpcedeOkcI1jpQmr91KgGGcyTjBKQm2JvD1rUVLVQfZRWyprpPTGji5fbz/7YFczsiqwkTD7DwYWK/nYKx49Df/mz281NsvwyM+BQnbIq0XujoyqMqql/gFEClvzZwKVL6XIkFhc4XVi9AX51wa0/A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=imdPfDew; arc=none smtp.client-ip=209.85.221.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="imdPfDew" Received: by mail-wr1-f54.google.com with SMTP id ffacd0b85a97d-3a365a6804eso1408754f8f.3 for ; Fri, 13 Jun 2025 07:05:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1749823529; x=1750428329; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DgbpV62LiWrQKsbfwnMZoUM9+oxW7YVl7WWXsQs3wV4=; b=imdPfDewX+YYI8XmW2g6UkWogRzy9XvGfaAwX7ZRJnCbyGT7LJy/fTrxTrteKICMOy WThfg0c1xtGbpp8JAVfRjk3FGkzqnjN8UDCWa1Rk8keGfnzfHYndjAeAuuL6X54ZDwMv y7GnNFsEw3svuyaGvuwT/U+7A70iNZ8yhL5UlJWJSe/XQ9lkU0DviTF5ZwAFOagYPAZd YnIqzWNw8SeycmoDVz8VcDYPaqhSWqtAHuEBHUg9aJVoSEjHVu5A3/tNDfChcY1tlZqQ WPkw6JNgeKMxwaT19MvzMwf68Tc/Wlgm2Sop2aW0s+SCM2eKoHp62/m4Byx1tcPodp6S rXxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749823529; x=1750428329; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DgbpV62LiWrQKsbfwnMZoUM9+oxW7YVl7WWXsQs3wV4=; b=d7K99eVAZOJMPZ7mQAbFuTDqBxCLZM/4G0+c01ysp48DrUmNSLa+NZ6+6QdWhgcnki +rMO3ZYWO5n9BAqmO60Tw4+8jtKvvYBN5QVkICjt4oyrhCvbX8bzEqOINa//m6QEYrt8 wr6AEjCUynlaVuZQ55bamISq8dpche2xXV968Qeei5JssNljvgppnWZGiBbRNSi2VOR2 gRzbzevHkgjG5Zg5NSOOx0cIvo/o4OCYbG+IymSscyN/jQHIMQJDV9RBkPhzmJUX6uKU 1bhSWT2GaXOkXIQHPEnr5ZeK3cUoj2TOI3giXAUHWS/lx/H1N6F+d7MXtl/aQrgVA84i 4o4w== X-Forwarded-Encrypted: i=1; AJvYcCUhHGbs+/W/ogHx1prPyDJB6IdeCx8v2xeBeoTZb64yFyWG1AFssCtbUHXMj++4JuZviN/UtbY/qL34oCI=@vger.kernel.org X-Gm-Message-State: AOJu0Yws7Rt42ZzauE3c2EdG3QW+2IvydsYf9QEH6kZ4KMapOZ7ByWjz EKDvq3LoCRrtYAkxG6MK7pqYkjKYrDUpJZwUqu/lb8SzhKvAIYcqKTwDbc+DnMAHv5o= X-Gm-Gg: ASbGncve20fEv/eEorX7PhjAKJREsgua98X0DXDNg+g3Q6WHvqju8qSUIA0UyarxWwo rY/ieubuiyfz2UT38mWJB8GcPYgIgzyv1H2XSKOA9LeqF10kp4aoIbB1KuoINaBgeDITxZ1OMY7 83v2aOjWl5FLV9/LeLTCQI9JU524LaqQx7ese/g1keaOLmXvzfP7Zdo3ZfTe6eEW0/tLa8vVBrd Lu9B6+9jrjaXZ76RUCKnZKEJpeNGmw+6UJSfHgwZcH+OnaxbEP9dX6/WKqiLbzCB4lWVhxZDUaw GiISAHMjEsQtig0FR/3I938AZV8YpgsEUGfOTJ20p2MatlMkR9QgpenClr47etX57CzIzogg1LT 3CJSr X-Google-Smtp-Source: AGHT+IH+uHLM8Q1BZumEUNO1NXYHlJbHUt903NUJo6OhYuVLmqYHugYn5cTTmIt9ZkzYIXnCM72nWg== X-Received: by 2002:a5d:5f8f:0:b0:3a4:d6ed:8df8 with SMTP id ffacd0b85a97d-3a568716312mr3152834f8f.39.1749823526913; Fri, 13 Jun 2025 07:05:26 -0700 (PDT) Received: from vingu-cube.. ([2a01:e0a:f:6020:f98:f3a4:ec28:1d4e]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3a568b27795sm2472276f8f.71.2025.06.13.07.05.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Jun 2025 07:05:26 -0700 (PDT) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, linux-kernel@vger.kernel.org Cc: Vincent Guittot Subject: [PATCH 1/4] sched/fair: Use protect_slice() instead of direct comparison Date: Fri, 13 Jun 2025 16:05:11 +0200 Message-ID: <20250613140514.2781138-2-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250613140514.2781138-1-vincent.guittot@linaro.org> References: <20250613140514.2781138-1-vincent.guittot@linaro.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Replace the test by the relevant protect_slice() function. Signed-off-by: Vincent Guittot Reviewed-by: Dhaval Giani (AMD) --- kernel/sched/fair.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 83157de5b808..44a09de38ddf 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1178,7 +1178,7 @@ static inline bool did_preempt_short(struct cfs_rq *c= fs_rq, struct sched_entity if (!sched_feat(PREEMPT_SHORT)) return false; =20 - if (curr->vlag =3D=3D curr->deadline) + if (protect_slice(curr)) return false; =20 return !entity_eligible(cfs_rq, curr); --=20 2.43.0 From nobody Fri Oct 10 13:38:09 2025 Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E96592E6D30 for ; Fri, 13 Jun 2025 14:05:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749823533; cv=none; b=FAUICSOjc68qp/9K8q8pmNPs8/AyfaPJK6kV3hxolXcbGzBvixcivurzz5fSiKJ7naEVaV36YEw5JCK8NrqadWcfVGGEnj7ColHuZngBNcfkf6Vet1xtHnKjh4pq5CxhS6W9Yym84y4r8biVNyCOk9gEMm7fuwnq9Us9wswzZr4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749823533; c=relaxed/simple; bh=2eXxHhED6PDOz4B5DoRIvFYZNAZy41feC2bruyyVAjk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oeydJ1hipGtSzze5xSwhRYmyxEqZuFaORS2F+VjgHi0BP9tCq0BqfCbzlKLPjv1uj9+HTZdNOeqg9f1SYhSkwuA/oNy0utvmiF9ELjivDLJA09NlR6uN2mE88PafDqfgxSAM2js6LlbWcRi6+kxbztO0dfC5wFY3cl/64EzhbMY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=a8vW2LFG; arc=none smtp.client-ip=209.85.221.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="a8vW2LFG" Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-3a53ee6fcd5so1417857f8f.1 for ; Fri, 13 Jun 2025 07:05:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1749823530; x=1750428330; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VlLCOC1uXTQGtq6J0ady+FM7lD87zFauRi/dRFmiIlk=; b=a8vW2LFGVH4yAhWRJn3ydEqM+UhObsoi8FOI6R6ou8jScHD0wuZ/RfmlKGgzSeNwcs 23W8bA02Bq9aQ8lz9o+RFP5W1p0E3jIwLIMBgnPv4sOrGSN1kr/z2BaY8Y7JmNvkIB9h x+mEV99xtT3NqP7qcM2VHT1jGla7gHelqJUrU57jylV7jgl15iSwLklCw4srJJZjTtnl TmP4xUy3/HTZUDeZvaIA07X2eSpTWOuUwEeXiIzmgBXjkGc2hsHuGlRllikNv/pFxyQa acnFq2wS4s8T2Xyk2r640B7jzgX6EQ4317dOUv2Qo/g6LXZ7OF6d/kRW9/JLkfV5S6wy 59wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749823530; x=1750428330; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VlLCOC1uXTQGtq6J0ady+FM7lD87zFauRi/dRFmiIlk=; b=xEL+3RRfPFBs6x544JFDy4STH7+N8Qg1pSYWPhsq+48geXdwgYQu/tAjj9gL3GNRzf XFigTxXhjrKKfXxhuNXCKM1R63oIKLTppho77iPAiAh7gJc4aHzVaofCITzuP4GL7Zsb bbDLl0owbEFZDp3ir5hMNKOA7KNA9gGILx0jeBtoKq5hJGKy+adbpojrXSjheAmWBXhb mb/hmdwFY81gu1ifLhWV2nTICxgLdPRYdVl49KTt9JgweHaqsQvhlarailCv73wJoYpM 4go688ov8sDFdiRUrMjJxHvohG4RKUN68pPWnA8yioHG/MmUTeAiJG5oP7lgyIa/Tndn fqVw== X-Forwarded-Encrypted: i=1; AJvYcCUiIV7gI9gKGqUWJEORGiMd4YKrjDLfR4sa1u6vSCURxZtlc+M6eiY1k8yTO+BZuG2ZIRSD0meR0dxXC+Q=@vger.kernel.org X-Gm-Message-State: AOJu0YzGNkK9OaEw7xftoaovd1E2JKCNoaywvvCVM2uJWdpU1QwYtdZQ 4PQN32fW3taHBn1X7uym3s8u1Z2DyHbtkkpbbRnkKH3YKC1GuL46q9uUD6fELYZJdW0= X-Gm-Gg: ASbGncuKqFD2kkG4TJQjKl6/w1t4Dyws2lGT/FpR5V5/1yHm75NFPSCS6LlYOigU/aa dDmHc1BaemufFuahGcOlGbJMxokZXldAfcN0rNBUSsAt7Dg8sq8+gjkAtpeUNO7WojCiE5G4bVE Lv5Eo1nJoruZmiKkDJZY4RZrX1HX+uDVD1LyPkJ+c6kt6jLqsb6We3thNrQrixgBRq5Vqszm1Pd 3bXtxRJJbDDYvYJZFJRcmNRQ9SVdYXDlW9Qar9bUuCEpwPzDXMUcosO27x6WNmKCfaxX/gX1ezd Ta79i2DlgZWM67m/OUCbKFCM3fnhvXM4KRHc/qibC+3MyJ/P7fcItuKte2wWjr3pTi1m4w== X-Google-Smtp-Source: AGHT+IF15AytVCT2tzFg06jLZDIADSIK1fHSnv83Zlv8R00kApPyS6NdCtGbJ6XFm30zsMZSvB8vQA== X-Received: by 2002:a05:6000:1ace:b0:3a4:f723:3e73 with SMTP id ffacd0b85a97d-3a5686709d3mr3065738f8f.16.1749823528506; Fri, 13 Jun 2025 07:05:28 -0700 (PDT) Received: from vingu-cube.. ([2a01:e0a:f:6020:f98:f3a4:ec28:1d4e]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3a568b27795sm2472276f8f.71.2025.06.13.07.05.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Jun 2025 07:05:27 -0700 (PDT) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, linux-kernel@vger.kernel.org Cc: Vincent Guittot Subject: [PATCH 2/4] sched/fair: Increase max lag clamping Date: Fri, 13 Jun 2025 16:05:12 +0200 Message-ID: <20250613140514.2781138-3-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250613140514.2781138-1-vincent.guittot@linaro.org> References: <20250613140514.2781138-1-vincent.guittot@linaro.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra sched_entity's lag is currently limited to the maximum between the tick and twice its slice. This is too short compared to the maximum custom slice that can be set and accumulated by other tasks. A task can accumulate up to its slice of negative lag while running to parity and the other runnable tasks can accumulate the same positive lag while waiting to run. This positive lag could be lost during dequeue when clamping it to twice task's slice if a task's slice is 100ms and others use a smaller value like the default 2.8ms. Clamp the lag of a task to the maximum slice of enqueued entities plus a tick as the update can be delayed to the next tick. Signed-off-by: Peter Zijlstra (Intel) [ Rebased and Fix max slice computation ] Signed-off-by: Vincent Guittot Reviewed-by: Dhaval Giani (AMD) --- include/linux/sched.h | 1 + kernel/sched/fair.c | 41 +++++++++++++++++++++++++++++++++++++---- 2 files changed, 38 insertions(+), 4 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 4f78a64beb52..89855ab45c43 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -576,6 +576,7 @@ struct sched_entity { u64 deadline; u64 min_vruntime; u64 min_slice; + u64 max_slice; =20 struct list_head group_node; unsigned char on_rq; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 44a09de38ddf..479b38dc307a 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -676,6 +676,8 @@ u64 avg_vruntime(struct cfs_rq *cfs_rq) return cfs_rq->min_vruntime + avg; } =20 +static inline u64 cfs_rq_max_slice(struct cfs_rq *cfs_rq); + /* * lag_i =3D S - s_i =3D w_i * (V - v_i) * @@ -689,17 +691,16 @@ u64 avg_vruntime(struct cfs_rq *cfs_rq) * EEVDF gives the following limit for a steady state system: * * -r_max < lag < max(r_max, q) - * - * XXX could add max_slice to the augmented data to track this. */ static void update_entity_lag(struct cfs_rq *cfs_rq, struct sched_entity *= se) { + u64 max_slice =3D cfs_rq_max_slice(cfs_rq) + TICK_NSEC; s64 vlag, limit; =20 WARN_ON_ONCE(!se->on_rq); =20 vlag =3D avg_vruntime(cfs_rq) - se->vruntime; - limit =3D calc_delta_fair(max_t(u64, 2*se->slice, TICK_NSEC), se); + limit =3D calc_delta_fair(max_slice, se); =20 se->vlag =3D clamp(vlag, -limit, limit); } @@ -795,6 +796,21 @@ static inline u64 cfs_rq_min_slice(struct cfs_rq *cfs_= rq) return min_slice; } =20 +static inline u64 cfs_rq_max_slice(struct cfs_rq *cfs_rq) +{ + struct sched_entity *root =3D __pick_root_entity(cfs_rq); + struct sched_entity *curr =3D cfs_rq->curr; + u64 max_slice =3D 0ULL; + + if (curr && curr->on_rq) + max_slice =3D curr->slice; + + if (root) + max_slice =3D max(max_slice, root->max_slice); + + return max_slice; +} + static inline bool __entity_less(struct rb_node *a, const struct rb_node *= b) { return entity_before(__node_2_se(a), __node_2_se(b)); @@ -820,6 +836,16 @@ static inline void __min_slice_update(struct sched_ent= ity *se, struct rb_node *n } } =20 +static inline void __max_slice_update(struct sched_entity *se, struct rb_n= ode *node) +{ + if (node) { + struct sched_entity *rse =3D __node_2_se(node); + + if (rse->max_slice > se->max_slice) + se->max_slice =3D rse->max_slice; + } +} + /* * se->min_vruntime =3D min(se->vruntime, {left,right}->min_vruntime) */ @@ -827,6 +853,7 @@ static inline bool min_vruntime_update(struct sched_ent= ity *se, bool exit) { u64 old_min_vruntime =3D se->min_vruntime; u64 old_min_slice =3D se->min_slice; + u64 old_max_slice =3D se->max_slice; struct rb_node *node =3D &se->run_node; =20 se->min_vruntime =3D se->vruntime; @@ -837,8 +864,13 @@ static inline bool min_vruntime_update(struct sched_en= tity *se, bool exit) __min_slice_update(se, node->rb_right); __min_slice_update(se, node->rb_left); =20 + se->max_slice =3D se->slice; + __max_slice_update(se, node->rb_right); + __max_slice_update(se, node->rb_left); + return se->min_vruntime =3D=3D old_min_vruntime && - se->min_slice =3D=3D old_min_slice; + se->min_slice =3D=3D old_min_slice && + se->max_slice =3D=3D old_max_slice; } =20 RB_DECLARE_CALLBACKS(static, min_vruntime_cb, struct sched_entity, @@ -852,6 +884,7 @@ static void __enqueue_entity(struct cfs_rq *cfs_rq, str= uct sched_entity *se) avg_vruntime_add(cfs_rq, se); se->min_vruntime =3D se->vruntime; se->min_slice =3D se->slice; + se->max_slice =3D se->slice; rb_add_augmented_cached(&se->run_node, &cfs_rq->tasks_timeline, __entity_less, &min_vruntime_cb); } --=20 2.43.0 From nobody Fri Oct 10 13:38:09 2025 Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D8D152E7643 for ; Fri, 13 Jun 2025 14:05:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749823536; cv=none; b=OcVC+EQiVTTa0M9A1JqXp3JcytSasYVPZKxauHQ5OMS4hGyBAA2rkjHYHcD45pk8MXBDL3yQH5nDe1pZJWwfxVHcKP1GrygKbOS79IenAlTrXTx80tZciJt1Xe3vZ0yIkfLc1rLLg0Fm4dy0QmI+/9lo8fOLadhfHLsunXw/+h0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749823536; c=relaxed/simple; bh=cdgNVRg3q+p4IrGkgXku/QNcy3g6z+UtJU2ARXzJxKg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SzPFa5Yw1aGjDLQr4Mt1dUAd3jZIy+8tpOjwFDBh0WT6YLY9tIPy99CH/S4RVMjM+4WIi88Frm7ZCLC/tBTgx2+wO4T6AyYkaWczg6/7EHKyri34lwcKh1wk24MTmsbBtYbsMO7P4j3FmlnpR9Y+w/LYjnVvOXxc00PhSY7V3i0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=VMHpcpm8; arc=none smtp.client-ip=209.85.128.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="VMHpcpm8" Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-450cfb79177so13259295e9.0 for ; Fri, 13 Jun 2025 07:05:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1749823533; x=1750428333; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0X2uv1HiFv0KbvzxwMCTug/zMsr1b3Ey/vikGScDg4A=; b=VMHpcpm8F3RWT3hcfgKRNPp/bHBu8QKQ/sIMGAQ7gycR1kltWM4/aaB5YtRlI0Svdr NZyoL137oJE+MYjc+gyIbJ4N/kDwUAG1IX5EUbE3gioZ/lOaBZEXgyuipZ2+/RFqtj/w R9eKH8iTzeIy43+NcyZu42sqN/KC3EGd0hCywC6g50+teahCDz95i032GhjmKVxUfxiO jf/O9yROeH7UhgBxrzGISTj2+kOJjaRKb9uYE3cjxGS1MlUWOcNbUl4moA2zI5s7sWz3 5aVuSSvU2k47PP7PgYv3bkWQPvMIxIDIMIGB3lRy973PhwIkojmMpg+9NJKM0BtxbE5k vcrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749823533; x=1750428333; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0X2uv1HiFv0KbvzxwMCTug/zMsr1b3Ey/vikGScDg4A=; b=dan9MXCVqDWoy6hJuaQkR1uDKvpNDo9NS+B6Hu1jCA1l2I6v3PyX14VyTHzT/HJlbX 25zy8CTwnT6/IKT0o6Q5nI/Xo2WUu7FUboAcrv4wtSQ73WBw8PQQnoa3Fu1nZqaYxfAp 4hdwavrjjG6Y/L/wHvbf3ia5IAZDgkZllZF0lMXb7uDveEP3xCZQPxqLUQFqOp1hlVYC Lc9YWcp3t0+4HkHpJLEgJVvT59HBLfsyp2vykj0Ks+Vo8vIDqXzRc+/HWk1B6iw10L69 M1K7WMfYu8yQ8/Wg9EWX9jJ5/V14eGhbGRXStXpju/TORPUHKT0UW2mIx/loHHhSDmeX RDnQ== X-Forwarded-Encrypted: i=1; AJvYcCX2nQXp95F4ijMssG75gxuoFsrZ0HEHGzz68lPItS1ZFF0QDZ2h9v0mQwXnxcnr9lcTfJbkjApMe+YqN9s=@vger.kernel.org X-Gm-Message-State: AOJu0YyS8ff5xhMuQEpvCt59J3oJqahzHJL6AeprzsNV62wsTh4Gh0N8 FjIh0LrHNafBuqoEuc/LdZrgSbmDb/yqlbnJJ/RgKmRnjGF3AoXcw0Ifq/tXsxRB49s= X-Gm-Gg: ASbGncteqPyHhIa0ckgd2m24otrT3Tr5GYxe/RLU6qNAEbbZSyoEJG+au7j6DUz0csz GyNl+RnXYJtaA6dK2p3jhIgw/yEYicNBC8Y8MrcagbesIbhYnhpyuaSyx5bdsYVlOUZFd7bCHGz G44UUeX+jDc2vGZOKdb3CrX1GG7Hu6d0TDcyMvabm1KUzT47crZoHofteTYi8tNFeXlv00TQMQL 4d17nwas35qZM3/HfWlyhUNDIjqy1HU9EpkZoIXNqIAouKvgHDflF6BfYgNwM4LZp+86Ne6vStr AEve6gpojYvd59eXiKJpKb/L1dvQ/V0HZSKpeLVTEzOqXlKnEqCqEmdyyUO9pe4lanEtHA== X-Google-Smtp-Source: AGHT+IHEQg7HeanKAzRdnE15qL/ZnvtjaZJ+CnD5r16xm2wOZZxjo7b8rbLUIvY0M3veNJmb+OIZ9Q== X-Received: by 2002:a05:600c:c4a5:b0:43d:46de:b0eb with SMTP id 5b1f17b1804b1-45334a979c1mr35195475e9.12.1749823533071; Fri, 13 Jun 2025 07:05:33 -0700 (PDT) Received: from vingu-cube.. ([2a01:e0a:f:6020:f98:f3a4:ec28:1d4e]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3a568b27795sm2472276f8f.71.2025.06.13.07.05.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Jun 2025 07:05:28 -0700 (PDT) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, linux-kernel@vger.kernel.org Cc: Vincent Guittot Subject: [PATCH 3/4] sched/fair: Limit run to parity to the min slice of enqueued entities Date: Fri, 13 Jun 2025 16:05:13 +0200 Message-ID: <20250613140514.2781138-4-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250613140514.2781138-1-vincent.guittot@linaro.org> References: <20250613140514.2781138-1-vincent.guittot@linaro.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Run to parity ensures that current will get a chance to run its full slice in one go but this can create large latency for entity with shorter slice that has alreasy exausted its slice and wait to run the next one. Clamp the run to parity duration to the shortest slice of all enqueued entities. Signed-off-by: Vincent Guittot --- kernel/sched/fair.c | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 479b38dc307a..d8345219dfd4 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -917,23 +917,32 @@ struct sched_entity *__pick_first_entity(struct cfs_r= q *cfs_rq) } =20 /* - * HACK, stash a copy of deadline at the point of pick in vlag, - * which isn't used until dequeue. + * HACK, Set the vruntime, up to which the entity can run before picking + * another one, in vlag, which isn't used until dequeue. + * In case of run to parity, we use the shortest slice of the enqueued + * entities. */ static inline void set_protect_slice(struct sched_entity *se) { - se->vlag =3D se->deadline; + u64 min_slice; + + min_slice =3D cfs_rq_min_slice(cfs_rq_of(se)); + + if (min_slice !=3D se->slice) + se->vlag =3D min(se->deadline, se->vruntime + calc_delta_fair(min_slice,= se)); + else + se->vlag =3D se->deadline; } =20 static inline bool protect_slice(struct sched_entity *se) { - return se->vlag =3D=3D se->deadline; + return ((s64)(se->vlag - se->vruntime) > 0); } =20 static inline void cancel_protect_slice(struct sched_entity *se) { if (protect_slice(se)) - se->vlag =3D se->deadline + 1; + se->vlag =3D se->vruntime; } =20 /* --=20 2.43.0 From nobody Fri Oct 10 13:38:09 2025 Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 43F382E764E for ; Fri, 13 Jun 2025 14:05:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749823538; cv=none; b=asRRV4cjFh25fNVbE3fIBAozRfQk+tcxxKOTw1PbISrXsqOomc1QAUhCzdB+eVu+9K1FATomuiKDbYMnNM0M5Qo7Vole3lrEm8aLeCYIlu7y4Doyow3ew7Y2ZkWBwR6xCJYu3Ly0NqZkpFiRGth/4g33GyvpxHK6L+VFhokxu7U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749823538; c=relaxed/simple; bh=zmOD1/rUzfz7QYBoFYJmCP01VLNfng0/qUUtfTajsEE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nl57+mEiDcQG9IlptbLVMn9UPlGnGkK7ZXmQwCuPXu9RfXJjnngLzm5Ums5H0pqlU5XRI8UQ5CAgCKwB7U9TCQHlSX1ehsRyfffm/ovpJVva8wISuMq0y9ddQjjAKooiRGBhP60DRGgOdNJ9HPG9WhgXyyRUYuWQbMFRnrOoxH0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=Y+OknngF; arc=none smtp.client-ip=209.85.128.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="Y+OknngF" Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-450ce3a2dd5so19780245e9.3 for ; Fri, 13 Jun 2025 07:05:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1749823534; x=1750428334; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xZq3HPM5M5NgUioHq85yF6dGPT1Zo3ahkiRvy5p1DrM=; b=Y+OknngFFnQ7PX356+X9TE0Rr4nS0mvkGpsZzuc5hdV6rYKGUfan46/1IRCBed0/pA 2P0Qm2V3xS7R39owVnLev/4JhJJrO9+jxiMksPms5PzJFLGG3CvuDxw5JXnJSN49gqFF kfEX77q16+DneJen7DPadgCCy7s9+hiJrzsGPj7ZCafH9mCT/b9ZlQ9Q6/SYumQGDZLo 3azlrwklboV4Q64zKlfCKb4ZPhqT797DNhfIhEOtuR8uEjDDuvGEglqCZM6/NvtINCzV Q1qWpSWCFo1FdJCKpNLwKY+l6OI1ZlmB5ba2wzdqx6GlzlAqjX3WIoE8MKfLGLZaxZyD r32Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749823534; x=1750428334; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xZq3HPM5M5NgUioHq85yF6dGPT1Zo3ahkiRvy5p1DrM=; b=UZK1OBTl06vvlEWun4tS1HAFXtAPTamu2zfxFKF233rsKQGIldFoCAHgt3xkpnvP8+ iosAaJMEWh+thaDYPnP78v1DmQ7bDK2vrobuYBsppDvAW9k2w7hBdKaRkLI3s2qLTiah f2/OahE0xaFGhn8BDtsxGfnkP1QvLDVbQvp207fgBmKKjxDcoacZaSFg3+dzf4XwQ+7s Qmio09mYvR0VACXT5V4ip42ArvD1skBT38tCZMLb3xDKmHJzq1ckMt48Zq+aCatX32W6 Ojn22oz41ZMQRTvyh1W7pclM3CPgDVa8+QtspjwxTQsgDH6Mu6x6oJeDKAikgPTstafR gX3Q== X-Forwarded-Encrypted: i=1; AJvYcCWcJy+3eEXr6PLsEQO027m7kH4UcS7gAq2qxQhyw3VkOuWlpZu5Oz4VWe44ogmLDrDZbUZk852ocgng8O8=@vger.kernel.org X-Gm-Message-State: AOJu0YyATI2/WEOCB+wFnIBilc9IVzKRiYMRQSbfn7Z2JC0+uyffjdte U2Rk6rIQDE8cFz5n41IjIJKhupirac+aFgsinP5qI3jsy/NYjm1qAI6w0I6r0TAOarU= X-Gm-Gg: ASbGncv5At/rwbu089ogL1YmFkL2DR8qfvWm61QUzFjDDXYedyRLJe0mPRGfSdtV5h1 uPnbPkKswesxpS/msrfTZEfwDM6NDevs9gp6p3pfzJBvDps/jUnwMbd7z5FJ2dtiJUgH1M31W9O 4tg5RyGHqMRAIiOsVfAf52LEzQB6jvicSNENS6SxMvs5pEDnONgC1MzsWcHAGJ2qCQ6gYsp124K nyjBeZxmduxqWZhWKAqjETBPod61kpdaGhNGIIwMA+kUEuluwgktONEh8wTvdGgxIVOuMcjFvsA 6YINWX+8/SB1Dd88ebOaegc4L+sm6Eo1ITUeedK28/keJFEMxrIeLyMczMcz5fIrbrAHkQ== X-Google-Smtp-Source: AGHT+IGJi/KZjt3wz1bY/aheeiTy+Pzp2o5DcLZKAfstFU+yQeDHcPOrBNopjtO43z6PODIO9wIcBA== X-Received: by 2002:a05:600c:1ca2:b0:43d:878c:7c40 with SMTP id 5b1f17b1804b1-45334ac986bmr41624025e9.10.1749823534008; Fri, 13 Jun 2025 07:05:34 -0700 (PDT) Received: from vingu-cube.. ([2a01:e0a:f:6020:f98:f3a4:ec28:1d4e]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3a568b27795sm2472276f8f.71.2025.06.13.07.05.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Jun 2025 07:05:33 -0700 (PDT) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, linux-kernel@vger.kernel.org Cc: Vincent Guittot Subject: [PATCH 4/4] sched/fair: Improve NO_RUN_TO_PARITY Date: Fri, 13 Jun 2025 16:05:14 +0200 Message-ID: <20250613140514.2781138-5-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250613140514.2781138-1-vincent.guittot@linaro.org> References: <20250613140514.2781138-1-vincent.guittot@linaro.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The slice protection of RUN_TO_PARITY should also be applied with a minimal quantum of time for NO_RUN_TO_PARITY in order to ensure a minimum runtime for each task with same slice duration but also to ensure periodic switch between threads. Signed-off-by: Vincent Guittot --- kernel/sched/fair.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d8345219dfd4..73bde511c53b 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -921,15 +921,20 @@ struct sched_entity *__pick_first_entity(struct cfs_r= q *cfs_rq) * one in vlag, which isn't used until dequeue. * In case of run to parity, we use the shortest slice of the enqueued * entities. + * When run to parity is disable we give a minimum quantum to the + * running entity to ensure progress. */ static inline void set_protect_slice(struct sched_entity *se) { - u64 min_slice; + u64 quantum; =20 - min_slice =3D cfs_rq_min_slice(cfs_rq_of(se)); + if (sched_feat(RUN_TO_PARITY)) + quantum =3D cfs_rq_min_slice(cfs_rq_of(se)); + else + quantum =3D min(se->slice, normalized_sysctl_sched_base_slice); =20 - if (min_slice !=3D se->slice) - se->vlag =3D min(se->deadline, se->vruntime + calc_delta_fair(min_slice,= se)); + if (quantum !=3D se->slice) + se->vlag =3D min(se->deadline, se->vruntime + calc_delta_fair(quantum, s= e)); else se->vlag =3D se->deadline; } @@ -981,7 +986,7 @@ static struct sched_entity *pick_eevdf(struct cfs_rq *c= fs_rq) if (curr && (!curr->on_rq || !entity_eligible(cfs_rq, curr))) curr =3D NULL; =20 - if (sched_feat(RUN_TO_PARITY) && curr && protect_slice(curr)) + if (curr && protect_slice(curr)) return curr; =20 /* Pick the leftmost entity if it's eligible */ @@ -1215,11 +1220,8 @@ static inline void update_curr_task(struct task_stru= ct *p, s64 delta_exec) cgroup_account_cputime(p, delta_exec); } =20 -static inline bool did_preempt_short(struct cfs_rq *cfs_rq, struct sched_e= ntity *curr) +static inline bool resched_next_quantum(struct cfs_rq *cfs_rq, struct sche= d_entity *curr) { - if (!sched_feat(PREEMPT_SHORT)) - return false; - if (protect_slice(curr)) return false; =20 @@ -1307,7 +1309,7 @@ static void update_curr(struct cfs_rq *cfs_rq) if (cfs_rq->nr_queued =3D=3D 1) return; =20 - if (resched || did_preempt_short(cfs_rq, curr)) { + if (resched || resched_next_quantum(cfs_rq, curr)) { resched_curr_lazy(rq); clear_buddies(cfs_rq, curr); } --=20 2.43.0