From nobody Fri Dec 19 14:05:47 2025 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BDEF5219E4 for ; Wed, 20 Dec 2023 00:19:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="3+mESWIH" Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-6d64b546e7eso1791598b3a.2 for ; Tue, 19 Dec 2023 16:19:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1703031583; x=1703636383; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Ma+3AbQ8bkO0LeZ0x4L+yPA6KTJo2YRmJIGPc7lQ3VM=; b=3+mESWIHpeKZxqJBcV8G9HW/poxO52h/+lJYw57Cw+JwIg8AiCH9EkSGdKRxUSsr17 7nSKmNY28xXTgcDlCoY7Nrg+BpWyasE1WfrpAynbj2RA6TpeK2AtcXEfmDrKvjhK8/sq Dd7fb2GzmOISaDnEtfVMAaBKaLaCs8+IfUwbjvkmQVNSVn2/EgKdWGHwsWpKUViSerPN RtPrWy3WhWwzQPnlOWGngEgC2QMXX3e0a9BmIIcsooy4Mb1FuRUs7MAECp972TxOxgJ7 beyUAqGwej9ya9JgdHX5HKkCbgeng4aGZE6rEJ/wQASJM5M7KSBh89xt6E746PvDwBAh E6RA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703031583; x=1703636383; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Ma+3AbQ8bkO0LeZ0x4L+yPA6KTJo2YRmJIGPc7lQ3VM=; b=EcycwKA9QGL2xb7+M16A0tPS6j4Wz3rrwhKfTI11wgK3TQYTdB8G4q5vZswGyDD17o E1Fmov0BNlZYV4PyqbeKLEqjJ5g4NSZClyIirAS1QmyEYhoM+j5HVHkTinnJMNtoly/h iOA1X4Max1L186AnXGvaY0MDJMkt6AqKYVSfMUH65qHCX3JfxGz8l5uzTiF/yrQpQeF7 CY/OEcO5pj8wEe67carb5jaJOzp9m6n7RIDzOZQYGUUiZ+Oq68tDCetsUz2VNva8suiO 0zPdBKwWFWDTvYlARIfi2gNun8HJHSWnofttqrf0gDs0cD4go1HDqCd/wspSLxQRt/KQ KdHg== X-Gm-Message-State: AOJu0YzQ64kiZQlhoMOks4Vy1TOkA7rkVc5TKjnEKLjCrWi4fpWIldP4 RB8Fo1ZHzybhbENrrI8/gGwqWyt/7ozdtPUlmFQvCfUsA/IOG4cSyqRPmz+ADIRjGtIBYTRFMR7 WcUgxiT8JXHTV9402X+wrcK6trP428QKnC7GNdhDuhNVoVDVNEQLu9vMLtVs87cRql15UPLo= X-Google-Smtp-Source: AGHT+IHi/7DMb6bi2lndxhzyg+zWbC/oUkKFAaeQms9i1nU22B4vG0I4/VrQBTIy0/6NHvOybXKkr3clE5f2 X-Received: from jstultz-noogler2.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:600]) (user=jstultz job=sendgmr) by 2002:aa7:8486:0:b0:6d8:a6fd:dc98 with SMTP id u6-20020aa78486000000b006d8a6fddc98mr81353pfn.6.1703031581731; Tue, 19 Dec 2023 16:19:41 -0800 (PST) Date: Tue, 19 Dec 2023 16:18:33 -0800 In-Reply-To: <20231220001856.3710363-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20231220001856.3710363-1-jstultz@google.com> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231220001856.3710363-23-jstultz@google.com> Subject: [PATCH v7 22/23] sched: Refactor dl/rt find_lowest/latest_rq logic From: John Stultz To: LKML Cc: John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Youssef Esmat , Mel Gorman , Daniel Bristot de Oliveira , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This pulls re-validation logic done in find_lowest_rq and find_latest_rq after re-acquiring the rq locks out into its own function. This allows us to later use a more complicated validation check for chain-migration when using proxy-exectuion. TODO: It seems likely we could consolidate this two functions further and leave the task_is_rt()/task_is_dl() checks externally? Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Youssef Esmat Cc: Mel Gorman Cc: Daniel Bristot de Oliveira Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: kernel-team@android.com Signed-off-by: John Stultz --- kernel/sched/deadline.c | 31 ++++++++++++++++++++----- kernel/sched/rt.c | 50 ++++++++++++++++++++++++++++------------- 2 files changed, 59 insertions(+), 22 deletions(-) diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 21e56ac58e32..8b5701727342 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -2172,6 +2172,30 @@ static int find_later_rq(struct task_struct *sched_c= tx, struct task_struct *exec return -1; } =20 +static inline bool dl_revalidate_rq_state(struct task_struct *task, struct= rq *rq, + struct rq *later) +{ + if (task_rq(task) !=3D rq) + return false; + + if (!cpumask_test_cpu(later->cpu, &task->cpus_mask)) + return false; + + if (task_on_cpu(rq, task)) + return false; + + if (!dl_task(task)) + return false; + + if (is_migration_disabled(task)) + return false; + + if (!task_on_rq_queued(task)) + return false; + + return true; +} + /* Locks the rq it finds */ static struct rq *find_lock_later_rq(struct task_struct *task, struct rq *= rq) { @@ -2204,12 +2228,7 @@ static struct rq *find_lock_later_rq(struct task_str= uct *task, struct rq *rq) =20 /* Retry if something changed. */ if (double_lock_balance(rq, later_rq)) { - if (unlikely(task_rq(task) !=3D rq || - !cpumask_test_cpu(later_rq->cpu, &task->cpus_mask) || - task_on_cpu(rq, task) || - !dl_task(task) || - is_migration_disabled(task) || - !task_on_rq_queued(task))) { + if (unlikely(!dl_revalidate_rq_state(task, rq, later_rq))) { double_unlock_balance(rq, later_rq); later_rq =3D NULL; break; diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index f8134d062fa3..fabb19891e95 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -1935,6 +1935,39 @@ static int find_lowest_rq(struct task_struct *sched_= ctx, struct task_struct *exe return -1; } =20 +static inline bool rt_revalidate_rq_state(struct task_struct *task, struct= rq *rq, + struct rq *lowest) +{ + /* + * We had to unlock the run queue. In + * the mean time, task could have + * migrated already or had its affinity changed. + * Also make sure that it wasn't scheduled on its rq. + * It is possible the task was scheduled, set + * "migrate_disabled" and then got preempted, so we must + * check the task migration disable flag here too. + */ + if (task_rq(task) !=3D rq) + return false; + + if (!cpumask_test_cpu(lowest->cpu, &task->cpus_mask)) + return false; + + if (task_on_cpu(rq, task)) + return false; + + if (!rt_task(task)) + return false; + + if (is_migration_disabled(task)) + return false; + + if (!task_on_rq_queued(task)) + return false; + + return true; +} + /* Will lock the rq it finds */ static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq = *rq) { @@ -1964,22 +1997,7 @@ static struct rq *find_lock_lowest_rq(struct task_st= ruct *task, struct rq *rq) =20 /* if the prio of this runqueue changed, try again */ if (double_lock_balance(rq, lowest_rq)) { - /* - * We had to unlock the run queue. In - * the mean time, task could have - * migrated already or had its affinity changed. - * Also make sure that it wasn't scheduled on its rq. - * It is possible the task was scheduled, set - * "migrate_disabled" and then got preempted, so we must - * check the task migration disable flag here too. - */ - if (unlikely(task_rq(task) !=3D rq || - !cpumask_test_cpu(lowest_rq->cpu, &task->cpus_mask) || - task_on_cpu(rq, task) || - !rt_task(task) || - is_migration_disabled(task) || - !task_on_rq_queued(task))) { - + if (unlikely(!rt_revalidate_rq_state(task, rq, lowest_rq))) { double_unlock_balance(rq, lowest_rq); lowest_rq =3D NULL; break; --=20 2.43.0.472.g3155946c3a-goog