From nobody Wed Dec 17 18:02:52 2025 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D7F4C15443F for ; Mon, 16 Dec 2024 03:12:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734318736; cv=none; b=Ha3aCwcCdA6N0oustlnCqffsCJO2JJM0dQxMbgBstmlp1nJUPjeNiR1nIjsNhUNigpg7GeVmpRM2UNQ6FOJH3XG6Hm5f1Ze+N1yBANAFyIG1mqt7OVxs/dOicT19Qf581zzeJFRehYXh295RYq4YcrruXt+uptlz0S2Iyt1qAe8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734318736; c=relaxed/simple; bh=TlSqGNDpjyEQ/sAS+WG9j8ZbPrrXCvTJG01tJ8bLoZ0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nqdOc9ZquQLia4/rsACFeWSwt5sxO/t5RZh549oqPfVSYontike66F7u8Qpy+eS9Fit2P0oTgosPFYRLyeajzesHsyjeD01RYb39HoDGxhQfzAXwrqTsNFylfvxrZv1lHFmMXnHGG3rvWm/jjeUHjAio7J6lkYtAor9PvBVoLuA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ibza3E0G; arc=none smtp.client-ip=209.85.210.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ibza3E0G" Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-728f1e66418so2947941b3a.2 for ; Sun, 15 Dec 2024 19:12:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734318734; x=1734923534; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ot9Lx1zg/XLL07layOuusQRDG2mU5e3t2AM5FahhmOs=; b=ibza3E0Gq21eowbZhy5dedKMhxeKt7mkHSvViGitZqCqBGh5l+L71BTjTxDcj0SQtv jLfcD9phyA8vTLjJxbQIJwrxNR5Ww1pKFYBeaVh1H6pgJRpH9OaA4hf37nKpH4SCRxqi KWjciwEyGdaA9xy1xuCynz3ITG9DeC6N6BDVZ5x8F21juWIqBHpEApwgkMKLrEfNAFg0 rDEVQK4dRs+8VA71SABFYIvkDI909yQtiPI8D4bTw1h8LsnIrLWYxy4r6zHo+QSpcEVq d3AGQsLGgMw2Ct2HZmSySjJdQ2ZwFl3eS4w+g93efOPM4Zf2JbpTFKZyfz0pQ1pFW8yt cCzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734318734; x=1734923534; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ot9Lx1zg/XLL07layOuusQRDG2mU5e3t2AM5FahhmOs=; b=UqEO1ms+bMtlrmytky+dKKCz1rF00TgpeG0zOZRc3vAjFFSDDGw3Ly4bcvo4P7nCew XsmlJA5RzFVgKX8qTBcBAlSgwiWhmtNmKOV51X6baLA7T9n9A6xH0pUB1/Ce8O3mw24l APpaY2tbthB+0g6sampBkG5Q4cKAn7JIQ0KtMM8bzOTmZZusyu7bOeiDvBTf9S9lQriC yKfRRlGKh2k62Z+w1FuA033kSuF3XZCDDTuCGIAh34xdcHnMQ+S3LDm4saG9NqYoDO0V RIW0yGpXjJjj4yJ3wE3DGKhFKfJkM9fXH8rsvaHBbFhGNLUPQWYgmDahcfWZWz74zwM0 t5dw== X-Forwarded-Encrypted: i=1; AJvYcCUIkfk8Hs7qKrExiBtLudhV3XtADVJo+jNepgFIFnvSu7/LMK+LrKrp0HLxTk87MazUmaoo3H7UqddiucE=@vger.kernel.org X-Gm-Message-State: AOJu0Yx3dvZ5Jn+E6z6LB6lgEudNKk4+xb8MIUnzKONNnmNc91TJXDY8 sAAlSepLTLI/gKv1dy3kbk/trUcEQdRPSzXWMcuKE2DgOQ1ig1cx X-Gm-Gg: ASbGncsC/Rk128hg+T6/GHTUd5UArVCOQNiNUB7lnWwxNG2L9GTEA1+S5ZGFB7FPZfZ 3VKuNr6EEn9MfRzsIVs/qcZnLZNHbYaItRSgBwcTSiUqkA5boSKv/pqEh6WQMeBZsIfp9Os35T6 HtA5Dx2Xxtj+8dOUCpaiEhJmHtB7993wjQ5xIvNf3sLXOjijVEuLWBv+vXjVl8T1TWgaKWOJ5g2 3H2QatcWObyIcq4cX9rW0/Qt/kSUC4m9TSt0UaDzwKAiQdUXKfR0neLsIC2eojdm1Vr6wT9oZN/ 435185yKKcUN X-Google-Smtp-Source: AGHT+IHNx85mGuRO8sIyUEFNPYeaTTw/FfXisv47Rm+pEH6iUsyoE+CZn8BvV/+JU0FIxkCuyAewqg== X-Received: by 2002:a05:6a20:3947:b0:1d9:3acf:8bdc with SMTP id adf61e73a8af0-1e1dfe6b5abmr18064863637.46.1734318734119; Sun, 15 Dec 2024 19:12:14 -0800 (PST) Received: from localhost ([58.29.143.236]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72925e6e8b1sm2172394b3a.139.2024.12.15.19.12.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 15 Dec 2024 19:12:13 -0800 (PST) From: Changwoo Min X-Google-Original-From: Changwoo Min To: tj@kernel.org, void@manifault.com, mingo@redhat.com, peterz@infradead.org, arighi@nvidia.com Cc: kernel-dev@igalia.com, linux-kernel@vger.kernel.org, Changwoo Min Subject: [PATCH v5 1/6] sched_ext: Relocate scx_enabled() related code Date: Mon, 16 Dec 2024 12:11:39 +0900 Message-ID: <20241216031144.98097-2-changwoo@igalia.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241216031144.98097-1-changwoo@igalia.com> References: <20241216031144.98097-1-changwoo@igalia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" scx_enabled() will be used in scx_rq_clock_update/invalidate() in the following patch, so relocate the scx_enabled() related code to the proper location. Signed-off-by: Changwoo Min --- kernel/sched/sched.h | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 76f5f53a645f..440ecedf871b 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1717,6 +1717,19 @@ struct rq_flags { =20 extern struct balance_callback balance_push_callback; =20 +#ifdef CONFIG_SCHED_CLASS_EXT +extern const struct sched_class ext_sched_class; + +DECLARE_STATIC_KEY_FALSE(__scx_ops_enabled); /* SCX BPF scheduler loaded */ +DECLARE_STATIC_KEY_FALSE(__scx_switched_all); /* all fair class tasks on S= CX */ + +#define scx_enabled() static_branch_unlikely(&__scx_ops_enabled) +#define scx_switched_all() static_branch_unlikely(&__scx_switched_all) +#else /* !CONFIG_SCHED_CLASS_EXT */ +#define scx_enabled() false +#define scx_switched_all() false +#endif /* !CONFIG_SCHED_CLASS_EXT */ + /* * Lockdep annotation that avoids accidental unlocks; it's like a * sticky/continuous lockdep_assert_held(). @@ -2505,19 +2518,6 @@ extern const struct sched_class rt_sched_class; extern const struct sched_class fair_sched_class; extern const struct sched_class idle_sched_class; =20 -#ifdef CONFIG_SCHED_CLASS_EXT -extern const struct sched_class ext_sched_class; - -DECLARE_STATIC_KEY_FALSE(__scx_ops_enabled); /* SCX BPF scheduler loaded */ -DECLARE_STATIC_KEY_FALSE(__scx_switched_all); /* all fair class tasks on S= CX */ - -#define scx_enabled() static_branch_unlikely(&__scx_ops_enabled) -#define scx_switched_all() static_branch_unlikely(&__scx_switched_all) -#else /* !CONFIG_SCHED_CLASS_EXT */ -#define scx_enabled() false -#define scx_switched_all() false -#endif /* !CONFIG_SCHED_CLASS_EXT */ - /* * Iterate only active classes. SCX can take over all fair tasks or be * completely disabled. If the former, skip fair. If the latter, skip SCX. --=20 2.47.1 From nobody Wed Dec 17 18:02:52 2025 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6B084189905 for ; Mon, 16 Dec 2024 03:12:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734318741; cv=none; b=OEAP7RjOEkwx330Q6IXhqRNokeYBOZFxZ9yFhQ0qxYLI8TJukNUgvW3mXzmZkRnG3MkrSJQtnpFCYWtcYVc/ZmCiJlwbUCTEeiOee1CGIgXS9hv7ktC0pROxq4hRQIkxPleirNG/anyDsLsVp3xw9Xm3qA7jADEf/FPjtb88EuM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734318741; c=relaxed/simple; bh=ARXOQdQSx11dZrIJgooSqC+dLcFILjDmqZ4pffr3ioU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oEdmSwLLWzZ4DgOYwEm4fb0Z/S7hASCha/9pTdnSr5J7crcoQUC3SeRCT9opSstr4ebv89aslcQ9iVTzL4sJEezIDa7uoxy+Z/ojg3rV2GDMog4ugTOGHeRJOw9bqC304EQj8kmL79on9UGE6/6uGiZZVngVAWiy9CSlY9Qy938= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=bRmO4C6W; arc=none smtp.client-ip=209.85.216.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bRmO4C6W" Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-2ef760a1001so3043339a91.0 for ; Sun, 15 Dec 2024 19:12:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734318739; x=1734923539; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZqkOMc7yUSchrOgOACYpXkIQ2LF3oE6mW3dZxc94vO8=; b=bRmO4C6WRZFLHNOieh2ibpIPJwwyUtxW3esW01gFq2JFLtR2cmz3ZBCcgU8Zj/FfHR 3C6LBCOWefWFGd3rbCBPxiVcweT2MfsufqhECMmaWlFgZW1D7TjVUXuNy6EE3GFC7kee uVv65gIbYrw4nCoQH8/sx+h/oCPuCATDdvgaRBfcCRSEwVhli711Ik7zGpETQ8JmZZQ+ F+0ieqZmpGPLR1zgLyJFJuq7Zmjagi3dNkmMSskoLDyXcny5UscOwEKZ4o/ariXS01/I s8e3dH0JGuUyV2pKOit/uSGwC44Q3byxNtR4d4Cu7pAFMhaG/Yp2PfFBVjLetWfcq2Kd nQwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734318739; x=1734923539; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZqkOMc7yUSchrOgOACYpXkIQ2LF3oE6mW3dZxc94vO8=; b=SHcVBo03V12NLvH6sCUu7pBq+th2wkdGeuJR4BoefUl8RraCyXgh89xXher/ruOX9V /Q3sFFYruYyPFbX2+ILW0/JfMIVuziAiT35QHNFkftwAothhdkgTztp7Nh9CL6WQxU18 O/6om1gjrWe4t6yke0V4Xx4kTHecyUJ+Pqfqj/serBdXpqB4UL5ijRQw3DB/qFLElaaJ Kl1dq18LwUwWBgIWH63+RAvfInhgyzfJvPTla3HXp0pyL7YjR6+JfS2pPeD+olGsJzby rI1FKukuWIprKLXaLkLVbDUbS/tvahCs+E7DZfyx2tFAqMyLvw6S8WfGsPXfJsyK9ERJ Zt1Q== X-Forwarded-Encrypted: i=1; AJvYcCXaFMmQIP5dMTfi0wIVr1Ei3/DK2iMVt8WBjIK/Jn2GgOyGMDEpMVMItLRFT3+q55zxRzvIVGzDGr3GT4k=@vger.kernel.org X-Gm-Message-State: AOJu0YzzT06OWJECuNkTJ60hWoWnhZYQ04kd5fTbeXmEsCrTkumVcKw9 GmK7R4TFPL8XBBf/TMhEyWaACfgsrAx9cNBwR1bSXOteNH8NpkZ+ X-Gm-Gg: ASbGncso1Ol7wIRxSuTkjZ4UfDJzAGRpNG9FhmYEBcU+cYT7XzjUppGYWHe+F0L3/eH R9SwLKY06RmXkZ6ghUgbuk7z3eTkmd7FvYRs/IP04NKtyCaYibbJ6xxwHmO0sYZwCSfnfrdy4te QfiTnDg40Ke8mrIWz5NdQOKSNI/pklguA4yfKh0d4pXAW73eTXi/Ug318iulGO5q0E6dkgdO0ch FSS8pTqUlKaNH0CakyuvqpOjJRUj3u0vpGVhZMuyyK1gPMJmDW8qUM5DDYkUfX+QvsyOXAWQwDm R7Rozv6UWEPj X-Google-Smtp-Source: AGHT+IFbo6wVuE1OCIlqpI8tH3ydH+JxuXb4lbrcE5BzLgtsPGBhrBLQ4qnmY9ZDoVjIApFmQRsZaw== X-Received: by 2002:a17:90b:3c0a:b0:2ee:c9b6:c266 with SMTP id 98e67ed59e1d1-2f28fb67310mr15938915a91.13.1734318738578; Sun, 15 Dec 2024 19:12:18 -0800 (PST) Received: from localhost ([58.29.143.236]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1e503cdsm32715075ad.113.2024.12.15.19.12.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 15 Dec 2024 19:12:18 -0800 (PST) From: Changwoo Min X-Google-Original-From: Changwoo Min To: tj@kernel.org, void@manifault.com, mingo@redhat.com, peterz@infradead.org, arighi@nvidia.com Cc: kernel-dev@igalia.com, linux-kernel@vger.kernel.org, Changwoo Min Subject: [PATCH v5 2/6] sched_ext: Implement scx_bpf_now_ns() Date: Mon, 16 Dec 2024 12:11:40 +0900 Message-ID: <20241216031144.98097-3-changwoo@igalia.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241216031144.98097-1-changwoo@igalia.com> References: <20241216031144.98097-1-changwoo@igalia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Returns a high-performance monotonically non-decreasing clock for the curre= nt CPU. The clock returned is in nanoseconds. It provides the following properties: 1) High performance: Many BPF schedulers call bpf_ktime_get_ns() frequently to account for execution time and track tasks' runtime properties. Unfortunately, in some hardware platforms, bpf_ktime_get_ns() -- which eventually reads a hardware timestamp counter -- is neither performant nor scalable. scx_bpf_now_ns() aims to provide a high-performance clock by using the rq clock in the scheduler core whenever possible. 2) High enough resolution for the BPF scheduler use cases: In most BPF scheduler use cases, the required clock resolution is lower than the most accurate hardware clock (e.g., rdtsc in x86). scx_bpf_now_ns() basically uses the rq clock in the scheduler core whenever it is valid. It considers that the rq clock is valid from the time the rq clock is updated (update_rq_clock) until the rq is unlocked (rq_unpin_lock). 3) Monotonically non-decreasing clock for the same CPU: scx_bpf_now_ns() guarantees the clock never goes backward when comparing them in the same CPU. On the other hand, when comparing clocks in different CPUs, there is no such guarantee -- the clock can go backward. It provides a monotonically *non-decreasing* clock so that it would provide the same clock values in two different scx_bpf_now_ns() calls in the same CPU during the same period of when the rq clock is valid. An rq clock becomes valid when it is updated using update_rq_clock() and invalidated when the rq is unlocked using rq_unpin_lock(). Let's suppose the following timeline in the scheduler core: T1. rq_lock(rq) T2. update_rq_clock(rq) T3. a sched_ext BPF operation T4. rq_unlock(rq) T5. a sched_ext BPF operation T6. rq_lock(rq) T7. update_rq_clock(rq) For [T2, T4), we consider that rq clock is valid (SCX_RQ_CLK_VALID is set), so scx_bpf_now_ns() calls during [T2, T4) (including T3) will return the rq clock updated at T2. For duration [T4, T7), when a BPF scheduler can still call scx_bpf_now_ns() (T5), we consider the rq clock is invalid (SCX_RQ_CLK_VALID is unset at T4). So when calling scx_bpf_now_ns() at T5, we will return a fresh clock value by calling sched_clock_cpu() internally. Also, to prevent getting outdated rq clocks from a previous scx scheduler, invalidate all the rq clocks when unloading a BPF scheduler. One example of calling scx_bpf_now_ns(), when the rq clock is invalid (like T5), is in scx_central [1]. The scx_central scheduler uses a BPF timer for preemptive scheduling. In every msec, the timer callback checks if the currently running tasks exceed their timeslice. At the beginning of the BPF timer callback (central_timerfn in scx_central.bpf.c), scx_central gets the current time. When the BPF timer callback runs, the rq clock could be invalid, the same as T5. In this case, scx_bpf_now_ns() returns a fresh clock value rather than returning the old one (T2). [1] https://github.com/sched-ext/scx/blob/main/scheds/c/scx_central.bpf.c Signed-off-by: Changwoo Min --- kernel/sched/core.c | 6 ++- kernel/sched/ext.c | 121 ++++++++++++++++++++++++++++++++++++++++++- kernel/sched/sched.h | 28 +++++++++- 3 files changed, 151 insertions(+), 4 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 95e40895a519..b0191977d919 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -789,6 +789,7 @@ static void update_rq_clock_task(struct rq *rq, s64 del= ta) void update_rq_clock(struct rq *rq) { s64 delta; + u64 clock; =20 lockdep_assert_rq_held(rq); =20 @@ -800,11 +801,14 @@ void update_rq_clock(struct rq *rq) SCHED_WARN_ON(rq->clock_update_flags & RQCF_UPDATED); rq->clock_update_flags |=3D RQCF_UPDATED; #endif + clock =3D sched_clock_cpu(cpu_of(rq)); + scx_rq_clock_update(rq, clock, true); =20 - delta =3D sched_clock_cpu(cpu_of(rq)) - rq->clock; + delta =3D clock - rq->clock; if (delta < 0) return; rq->clock +=3D delta; + update_rq_clock_task(rq, delta); } =20 diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c index afeed9dfeecb..991a86e87187 100644 --- a/kernel/sched/ext.c +++ b/kernel/sched/ext.c @@ -4910,7 +4910,7 @@ static void scx_ops_disable_workfn(struct kthread_wor= k *work) struct task_struct *p; struct rhashtable_iter rht_iter; struct scx_dispatch_q *dsq; - int i, kind; + int i, kind, cpu; =20 kind =3D atomic_read(&scx_exit_kind); while (true) { @@ -4993,6 +4993,15 @@ static void scx_ops_disable_workfn(struct kthread_wo= rk *work) scx_task_iter_stop(&sti); percpu_up_write(&scx_fork_rwsem); =20 + /* + * Invalidate all the rq clocks to prevent getting outdated + * rq clocks from a previous scx scheduler. + */ + for_each_possible_cpu(cpu) { + struct rq *rq =3D cpu_rq(cpu); + scx_rq_clock_invalidate(rq); + } + /* no task is on scx, turn off all the switches and flush in-progress cal= ls */ static_branch_disable(&__scx_ops_enabled); for (i =3D SCX_OPI_BEGIN; i < SCX_OPI_END; i++) @@ -7601,6 +7610,115 @@ __bpf_kfunc struct cgroup *scx_bpf_task_cgroup(stru= ct task_struct *p) } #endif =20 +/** + * scx_bpf_now_ns - Returns a high-performance monotonically non-decreasing + * clock for the current CPU. The clock returned is in nanoseconds. + * + * It provides the following properties: + * + * 1) High performance: Many BPF schedulers call bpf_ktime_get_ns() freque= ntly + * to account for execution time and track tasks' runtime properties. + * Unfortunately, in some hardware platforms, bpf_ktime_get_ns() -- which + * eventually reads a hardware timestamp counter -- is neither performant= nor + * scalable. scx_bpf_now_ns() aims to provide a high-performance clock by + * using the rq clock in the scheduler core whenever possible. + * + * 2) High enough resolution for the BPF scheduler use cases: In most BPF + * scheduler use cases, the required clock resolution is lower than the m= ost + * accurate hardware clock (e.g., rdtsc in x86). scx_bpf_now_ns() basical= ly + * uses the rq clock in the scheduler core whenever it is valid. It consi= ders + * that the rq clock is valid from the time the rq clock is updated + * (update_rq_clock) until the rq is unlocked (rq_unpin_lock). + * + * 3) Monotonically non-decreasing clock for the same CPU: scx_bpf_now_ns() + * guarantees the clock never goes backward when comparing them in the sa= me + * CPU. On the other hand, when comparing clocks in different CPUs, there + * is no such guarantee -- the clock can go backward. It provides a + * monotonically *non-decreasing* clock so that it would provide the same + * clock values in two different scx_bpf_now_ns() calls in the same CPU + * during the same period of when the rq clock is valid. + */ +__bpf_kfunc u64 scx_bpf_now_ns(void) +{ + struct rq *rq; + u64 clock; + + preempt_disable(); + + /* + * If the rq clock is valid, use the cached rq clock + * whenever the clock does not go backward. + */ + rq =3D this_rq(); + clock =3D READ_ONCE(rq->scx.clock); + + if (!(READ_ONCE(rq->scx.flags) & SCX_RQ_CLK_VALID) || + time_after_eq64(READ_ONCE(rq->scx.prev_clock), clock)) { + /* + * If the rq clock is invalid or goes backward, + * start a new rq clock period with a fresh sched_clock_cpu(). + * + * The cached rq clock can go backward because this kfunc can + * be called from anywhere, including timer interrupts and + * tracepoints for code running in the IRQ context. + * + * For example, suppose a timer interrupt occurred while + * running scx_bpf_now_ns() *after* reading the rq clock and + * *before* comparing the if condition. + * + * The timer interrupt will eventually call a BPF scheduler's + * ops.tick(), and the BPF scheduler can call scx_bpf_now_ns(). + * Since the scheduler core updates the rq clock before calling + * ops.tick(), the scx_bpf_now_ns() call will get the fresh + * clock. + * + * After handling the timer interrupt, the interrupted + * scx_bpf_now_ns() will be resumed, so the if condition will + * be compared. + * + * In this case, the clock read before the timer interrupt + * will be the same as rq->scx.prev_clock. + * + * When such a case is detected (i.e., prev_clock is smaller + * or equal to the current clock), get a new rq clock with a + * fresh sched_clock_cpu(). + * + * The following illustrates the detailed sequence over time: + * + * Time| Task | Timer interrupt + * ----+------------------------+------------------------------ + * T1 | call scx_bpf_now_ns() | + * T2 | load rq->scx.clock | + * T3 | compare rq->scx.flags | + * T4 | | timer interrupt occured + * T5 | | sched_tick() called + * T6 | | + rq_lock() called + * T7 | | + update_rq_clock() called + * T8 | | + sched_class->task_tick() + * T9 | | + scx_bpf_now_ns() returns + * | | a fresh clock updated at T7. + * T10 | | + rq_unlock() called + * T11 | resume the execution | + * T12 | compare scx.prev_clock | + * + * Note that since the scx.prev_clock at T12 was updated at T7, + * it will be the same as rq->scx.clock at T2. + */ + clock =3D sched_clock_cpu(cpu_of(rq)); + + /* + * The rq clock is updated outside of the rq lock. + * In this case, keep the updated rq clock invalid so the next + * kfunc call outside the rq lock gets a fresh rq clock. + */ + scx_rq_clock_update(rq, clock, false); + } + + preempt_enable(); + + return clock; +} + __bpf_kfunc_end_defs(); =20 BTF_KFUNCS_START(scx_kfunc_ids_any) @@ -7632,6 +7750,7 @@ BTF_ID_FLAGS(func, scx_bpf_cpu_rq) #ifdef CONFIG_CGROUP_SCHED BTF_ID_FLAGS(func, scx_bpf_task_cgroup, KF_RCU | KF_ACQUIRE) #endif +BTF_ID_FLAGS(func, scx_bpf_now_ns) BTF_KFUNCS_END(scx_kfunc_ids_any) =20 static const struct btf_kfunc_id_set scx_kfunc_set_any =3D { diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 440ecedf871b..892975543a6d 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -754,6 +754,7 @@ enum scx_rq_flags { SCX_RQ_BAL_PENDING =3D 1 << 2, /* balance hasn't run yet */ SCX_RQ_BAL_KEEP =3D 1 << 3, /* balance decided to keep current */ SCX_RQ_BYPASSING =3D 1 << 4, + SCX_RQ_CLK_VALID =3D 1 << 5, /* RQ clock is fresh and valid */ =20 SCX_RQ_IN_WAKEUP =3D 1 << 16, SCX_RQ_IN_BALANCE =3D 1 << 17, @@ -766,9 +767,11 @@ struct scx_rq { unsigned long ops_qseq; u64 extra_enq_flags; /* see move_task_to_local_dsq() */ u32 nr_running; - u32 flags; u32 cpuperf_target; /* [0, SCHED_CAPACITY_SCALE] */ bool cpu_released; + u32 flags; + u64 clock; /* current per-rq clock -- see scx_bpf_now_ns() */ + u64 prev_clock; /* previous per-rq clock -- see scx_bpf_now_ns() */ cpumask_var_t cpus_to_kick; cpumask_var_t cpus_to_kick_if_idle; cpumask_var_t cpus_to_preempt; @@ -1725,9 +1728,30 @@ DECLARE_STATIC_KEY_FALSE(__scx_switched_all); /* all= fair class tasks on SCX */ =20 #define scx_enabled() static_branch_unlikely(&__scx_ops_enabled) #define scx_switched_all() static_branch_unlikely(&__scx_switched_all) + +static inline void scx_rq_clock_update(struct rq *rq, u64 clock, bool vali= d) +{ + if (!scx_enabled()) + return; + WRITE_ONCE(rq->scx.prev_clock, rq->scx.clock); + WRITE_ONCE(rq->scx.clock, clock); + if (valid) + WRITE_ONCE(rq->scx.flags, rq->scx.flags | SCX_RQ_CLK_VALID); +} + +static inline void scx_rq_clock_invalidate(struct rq *rq) +{ + if (!scx_enabled()) + return; + WRITE_ONCE(rq->scx.flags, rq->scx.flags & ~SCX_RQ_CLK_VALID); +} + #else /* !CONFIG_SCHED_CLASS_EXT */ #define scx_enabled() false #define scx_switched_all() false + +static inline void scx_rq_clock_update(struct rq *rq, u64 clock, bool vali= d) {} +static inline void scx_rq_clock_invalidate(struct rq *rq) {} #endif /* !CONFIG_SCHED_CLASS_EXT */ =20 /* @@ -1759,7 +1783,7 @@ static inline void rq_unpin_lock(struct rq *rq, struc= t rq_flags *rf) if (rq->clock_update_flags > RQCF_ACT_SKIP) rf->clock_update_flags =3D RQCF_UPDATED; #endif - + scx_rq_clock_invalidate(rq); lockdep_unpin_lock(__rq_lockp(rq), rf->cookie); } =20 --=20 2.47.1 From nobody Wed Dec 17 18:02:52 2025 Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C40F719258B for ; Mon, 16 Dec 2024 03:12:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734318744; cv=none; b=XKeldMzTNfTdiboF4DidaSF8TlsAO/c//CEL6w/Fb2tc91HCLGNkrTsUnn9wV5Hp4GU9HBHQ2XAe+wHaC6nRbt9XJTFp/0auUSjGZzx79TF4u/XxSSB5IeFU5uKk9iD+OHRT3YtyguI9Gsb4LASwHj1oYirW+ej4bahPf93XVDU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734318744; c=relaxed/simple; bh=GZDtGSP3knc4XfhDJndoAF3aAtv9DQSb0onbM9mot6Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hq7cNy4Koz++6d5BpAJecXVn3X7Frd/dPZSNGFazjTRT1kNiD3d0AyZawo47MIafz35NNyM4kwDzz5Gr5Q+pBYREkO8iKU5iTVSMOjuNEXYQYmdm9lQ5OqXhK51l1Qc9DpEacqL83n97FLZbArJcV+pSJVjkRlgX7V7W3d8LKx8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=P1OWYE1N; arc=none smtp.client-ip=209.85.210.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="P1OWYE1N" Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-728f28744c5so3111724b3a.1 for ; Sun, 15 Dec 2024 19:12:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734318742; x=1734923542; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3qwdnhpnNZ/kDLebmCqGGOAqoWdsfoXXWVd5XUHDoNY=; b=P1OWYE1N6A6si4wPljkyOY6rSknFQ2XTH4cdMBxqiFINzD2IQt6dVSzDIAB9QHe+F1 ePlm0Vx75Xo4IrPdentab8KtPxoYlSfXKx6vDI7z7hQv/Nj3mFKbJNLL9Wfx2qDwmUMy 68qO8kYErkXttG+o2eFyxJTY8d0l3dpCiFy09VbS1fPqEl77PcrUEdafe6g/+gT9g973 NCYrAedz3ke2jAraas3+FOZ0r6O8hP5QuPC3s4WC3VB9UUS0oAUZGcOlslY2PXG3Atp4 I2A4p3pmCYJia+Rao3ovExiGU9GM6CsLK/SZHzU9OePGu4Z6om85C9IQVbNxXvL/bby+ j1Og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734318742; x=1734923542; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3qwdnhpnNZ/kDLebmCqGGOAqoWdsfoXXWVd5XUHDoNY=; b=U1qN4T5Oszn9TDrgvIY9n7ncb+tx1bOPk0SjeYdJpe1iumKlRHryLeh/02BDF7KWN4 jEcOspTOLmsNnRxNb5UCINAKh3aRzNHGgTuMHzrEfhbaHvaH0Jlvy8c2f3YizmUvZ1ha k6m5IYWhZXMJi2MH/YHv4MqwHUa24/t3WC9SApjLI9nwaP/SNW6/Vo+EDW1E3qxgzg0Z pJKC7FNxRl4yEvFSW1vWppRvpMyspj0cjx0Wig57nmxDpS2WEIVg3wUAXeDU4u6rFT24 g5iXoCTM9tv5jiX43RF1rujbeV4e04NXHfRGnOfUEFouZShuoWmZgtZ6WZ4LoYpy1twp Cfbg== X-Forwarded-Encrypted: i=1; AJvYcCW+Yh/90kDfPAob42yISLn33F7koi2rpnhhbOVgILhlsTRcLdWsnkXPIR5ZSsuWp00WX0w7mYZu/DO11H8=@vger.kernel.org X-Gm-Message-State: AOJu0Yw+BvaoAAQsdrF7vrk9sWzKICMSSsf/v17/F1ASo/FPyC3IhySx 4TUtb5Emr7jBxRJUYFeE9SyukauoaSDXw8bZ6gSFCyqSAOMqw7lc X-Gm-Gg: ASbGnct4zv2ekORKCw7jSHY/wJN6hTpU3x1d5Nr+GzSCFkZD2dehSN2jLeiOKavAo/g SS0gEFcAk3P3M5GnzS6M1hW/lwlK4UWW0ZFS8hP1LLQ3nwA0S2SZCI34U32b9qDA1w/HBXzV5eN cqI7vzWpgiYdzNQOj8to0Louj0qO8T86zEtQVvWr00oWhTX46bMH4YfjS9KaolEJZpf47+ytMD5 7jzjcJ2oOgSiTUwQwG+kkykdYZRphjYqEl/88SpuUl6WOhu2jH8H/Cso+U64OQ3+WVwglb8NXra 13dyB91LP7gb X-Google-Smtp-Source: AGHT+IECXXkjMF2cE+MK1ecDOadXetISrI342qnY5QQmSTp/jMki0OfIC4i3ujBXJzrYC7ApnXLdGg== X-Received: by 2002:a05:6a00:188c:b0:725:cfa3:bc6b with SMTP id d2e1a72fcca58-7290c0de7b8mr16346546b3a.3.1734318741945; Sun, 15 Dec 2024 19:12:21 -0800 (PST) Received: from localhost ([58.29.143.236]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72918ad55dbsm3650518b3a.43.2024.12.15.19.12.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 15 Dec 2024 19:12:21 -0800 (PST) From: Changwoo Min X-Google-Original-From: Changwoo Min To: tj@kernel.org, void@manifault.com, mingo@redhat.com, peterz@infradead.org, arighi@nvidia.com Cc: kernel-dev@igalia.com, linux-kernel@vger.kernel.org, Changwoo Min Subject: [PATCH v5 3/6] sched_ext: Add scx_bpf_now_ns() for BPF scheduler Date: Mon, 16 Dec 2024 12:11:41 +0900 Message-ID: <20241216031144.98097-4-changwoo@igalia.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241216031144.98097-1-changwoo@igalia.com> References: <20241216031144.98097-1-changwoo@igalia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" scx_bpf_now_ns() is added to the header files so the BPF scheduler can use it. Signed-off-by: Changwoo Min --- tools/sched_ext/include/scx/common.bpf.h | 1 + tools/sched_ext/include/scx/compat.bpf.h | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/tools/sched_ext/include/scx/common.bpf.h b/tools/sched_ext/inc= lude/scx/common.bpf.h index 858ba1f438f6..79f0798a5350 100644 --- a/tools/sched_ext/include/scx/common.bpf.h +++ b/tools/sched_ext/include/scx/common.bpf.h @@ -76,6 +76,7 @@ bool scx_bpf_task_running(const struct task_struct *p) __= ksym; s32 scx_bpf_task_cpu(const struct task_struct *p) __ksym; struct rq *scx_bpf_cpu_rq(s32 cpu) __ksym; struct cgroup *scx_bpf_task_cgroup(struct task_struct *p) __ksym __weak; +u64 scx_bpf_now_ns(void) __ksym __weak; =20 /* * Use the following as @it__iter when calling scx_bpf_dsq_move[_vtime]() = from diff --git a/tools/sched_ext/include/scx/compat.bpf.h b/tools/sched_ext/inc= lude/scx/compat.bpf.h index d56520100a26..2f16f30ae741 100644 --- a/tools/sched_ext/include/scx/compat.bpf.h +++ b/tools/sched_ext/include/scx/compat.bpf.h @@ -125,6 +125,11 @@ bool scx_bpf_dispatch_vtime_from_dsq___compat(struct b= pf_iter_scx_dsq *it__iter, false; \ }) =20 +#define scx_bpf_now_ns() \ + (bpf_ksym_exists(scx_bpf_now_ns) ? \ + scx_bpf_now_ns() : \ + bpf_ktime_get_ns()) + /* * Define sched_ext_ops. This may be expanded to define multiple variants = for * backward compatibility. See compat.h::SCX_OPS_LOAD/ATTACH(). --=20 2.47.1 From nobody Wed Dec 17 18:02:52 2025 Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 391EA1BBBD7 for ; Mon, 16 Dec 2024 03:12:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734318747; cv=none; b=Sbr2pQksbIM1621ArXRKQr87NXusgXNzlHD3yc4eU9DYP1jTaOYaLteU1lnh2f0p+GJKdpxXEXMHyl888kmlQhFkMNJe9X3NhxGErnjrv8DdjJopZN46jeF4Wd3Kf5e8yVogdWFIaYsoWeVk4sHUeeWdub1e4Wnf/Ga7fJXG+Lg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734318747; c=relaxed/simple; bh=tw/sZ6xyV0Pw2jZXspaHJWiWfZ9xSORxU2ivpIey0fc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IjyuU763bCXB2me+TQL7xZ1X/JLvdyF1TEUO1X3F4YvefNQDIwzCR085tV/nrg5Y5ienHE0hqCHqrcNTAatpRY+8iSvlo50zJyQkOTty0MyUlMP/tFdYK1FcD6/005dPTH4Cq2YiKbc/IyZ2GpAYhPWi1XwA21S02ekWztG4IfE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Q0M3sZ+4; arc=none smtp.client-ip=209.85.210.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Q0M3sZ+4" Received: by mail-pf1-f179.google.com with SMTP id d2e1a72fcca58-725dc290c00so3556415b3a.0 for ; Sun, 15 Dec 2024 19:12:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734318745; x=1734923545; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=K16k0CtGst/uGn1z/pOeYRcdvDetSUcyUB4347UZ9Ic=; b=Q0M3sZ+4JJaOdUD1yurVJpg+DkL1NHgM2OO7Fagg9LL6mj9bdXTzGDCK8FzpaGi8eD i6dCeDqqkPduHTGslPZWxAQyzOxMDTNxRtEWSXFj+PxlrXiZKcmIEx0/p9MRcnD4ocgw 8mD3ap/YhPOKQkl7w7zYIGBwhtI6W/aLbjiXCNk+l9Y3BpE8bf4HeDJ+PqraR4BhT8hm seqeRTfp1QM+NjmNDJIb7dWkFQ7Gk+C27HoWknZ1vSzT3FdF3Ogc1VlNVg/rfXp6Pflq G2dySHek9Yun9mkTEv7dssCTzl/BrBPml6qfIyknPOpUcAwMUjP2k6++CQ6xGD1d1+9R kPRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734318745; x=1734923545; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=K16k0CtGst/uGn1z/pOeYRcdvDetSUcyUB4347UZ9Ic=; b=wURhqimjLMGYVsvJ4Pe/Q2XVbTzC4rcLX3HzOmsQzTb+xSRrxwBaUq7KTqv0slXqFm hrgXBh3OZ0YWYIBCxt9Sn87ocHMUEhZ01KRWgkHfSemplo3xgFhBVcCv2AbzDq++b3iA XImw5WLwm+HqTqEoR7PbGpbf2WvEeAstssin4Eu4Fhkuc64ZNgB1rwURZh2s3vT41luA 3dd/nAB0kISZgqZ/k1RJZ6JZrOMLP1MejaFHFuXfbA/Zt7qfEK7J9U09hmAn1QgCAhBR bavQkBmk9Lo+ucTW5F1AAFnTNml0ySxiuZde7lHe7RJLxjma5nwYSHM+jbnyA0h/vPdr Bi3Q== X-Forwarded-Encrypted: i=1; AJvYcCUM9OVLzN/Cnq/6MiOEZGKhHLz3hcYfjh2A2ftp82kRrgjgW66XafL5E2Av7+qcddmrcD+YcNniqhFJUx8=@vger.kernel.org X-Gm-Message-State: AOJu0YwlLOENCVDberXv/lLf0YFbNZH1a3kuTgk7mp/N6S7e6eeQzl3V E04z32+i7bUFuamwwXqhJqmmedC4fnuVfVY+gibrGyg9INqSWFiU X-Gm-Gg: ASbGncvOK98IlRRA4RA5aIz7BnKrltlUgRtdYsnrS149h742DVXs77F0RVUhMFujfsL osn7/3mHB/xdSnefSfsUpJM2noIj2NOY/Pe8ComGs9k1MXSuuOJvegSR18A3fEVsztWsPWxO/w+ LQEER1n7f3SzeGjKhpXwqtLNNrmX09GDAWpxVg/R3Rz69hFNOepUIJApllyCKspKbhFOMFaHuDt Co9uH+P1/kM5SLFwEdM3Uv8akSHeeTa2EUUeOmdUYx2fomnuKCaS/lESwJgAZYISC+MWQyE46qi 1MDoXjhOuQ12 X-Google-Smtp-Source: AGHT+IGCDjtPx3t6cmLqhK9W8kHDOpUk/3/BG6YGZkhjZpTzZqklF78LRMOz3NhFsVDK0VClUQmyQQ== X-Received: by 2002:a05:6a20:a11c:b0:1e1:e1c0:1c05 with SMTP id adf61e73a8af0-1e1e1c0202amr15348954637.9.1734318745406; Sun, 15 Dec 2024 19:12:25 -0800 (PST) Received: from localhost ([58.29.143.236]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-801d5ac8766sm3186322a12.33.2024.12.15.19.12.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 15 Dec 2024 19:12:25 -0800 (PST) From: Changwoo Min X-Google-Original-From: Changwoo Min To: tj@kernel.org, void@manifault.com, mingo@redhat.com, peterz@infradead.org, arighi@nvidia.com Cc: kernel-dev@igalia.com, linux-kernel@vger.kernel.org, Changwoo Min Subject: [PATCH v5 4/6] sched_ext: Add time helpers for BPF schedulers Date: Mon, 16 Dec 2024 12:11:42 +0900 Message-ID: <20241216031144.98097-5-changwoo@igalia.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241216031144.98097-1-changwoo@igalia.com> References: <20241216031144.98097-1-changwoo@igalia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The following functions are added for BPF schedulers: - vtime_delta(after, before) - vtime_after(a, b) - vtime_before(a, b) - vtime_after_eq(a, b) - vtime_before_eq(a, b) - vtime_in_range(a, b, c) - vtime_in_range_open(a, b, c) Signed-off-by: Changwoo Min --- tools/sched_ext/include/scx/common.bpf.h | 94 ++++++++++++++++++++++++ 1 file changed, 94 insertions(+) diff --git a/tools/sched_ext/include/scx/common.bpf.h b/tools/sched_ext/inc= lude/scx/common.bpf.h index 79f0798a5350..923bbf57e4f1 100644 --- a/tools/sched_ext/include/scx/common.bpf.h +++ b/tools/sched_ext/include/scx/common.bpf.h @@ -408,6 +408,100 @@ static __always_inline const struct cpumask *cast_mas= k(struct bpf_cpumask *mask) void bpf_rcu_read_lock(void) __ksym; void bpf_rcu_read_unlock(void) __ksym; =20 +/* + * Time helpers, most of which are from jiffies.h. + */ + +/** + * vtime_delta - Calculate the delta between new and old time stamp + * @after: first comparable as u64 + * @before: second comparable as u64 + * + * Return: the time difference, which is >=3D 0 + */ +static inline s64 vtime_delta(u64 after, u64 before) +{ + return (s64)(after - before) > 0 ? : 0; +} + +/** + * vtime_after - returns true if the time a is after time b. + * @a: first comparable as u64 + * @b: second comparable as u64 + * + * Do this with "<0" and ">=3D0" to only test the sign of the result. A + * good compiler would generate better code (and a really good compiler + * wouldn't care). Gcc is currently neither. + * + * Return: %true is time a is after time b, otherwise %false. + */ +static inline bool vtime_after(u64 a, u64 b) +{ + return (s64)(b - a) < 0; +} + +/** + * vtime_before - returns true if the time a is before time b. + * @a: first comparable as u64 + * @b: second comparable as u64 + * + * Return: %true is time a is before time b, otherwise %false. + */ +static inline bool vtime_before(u64 a, u64 b) +{ + return vtime_after(b, a); +} + +/** + * vtime_after_eq - returns true if the time a is after or the same as tim= e b. + * @a: first comparable as u64 + * @b: second comparable as u64 + * + * Return: %true is time a is after or the same as time b, otherwise %fals= e. + */ +static inline bool vtime_after_eq(u64 a, u64 b) +{ + return (s64)(a - b) >=3D 0; +} + +/** + * vtime_before_eq - returns true if the time a is before or the same as t= ime b. + * @a: first comparable as u64 + * @b: second comparable as u64 + * + * Return: %true is time a is before or the same as time b, otherwise %fal= se. + */ +static inline bool vtime_before_eq(u64 a, u64 b) +{ + return vtime_after_eq(b, a); +} + +/** + * vtime_in_range - Calculate whether a is in the range of [b, c]. + * @a: time to test + * @b: beginning of the range + * @c: end of the range + * + * Return: %true is time a is in the range [b, c], otherwise %false. + */ +static inline bool vtime_in_range(u64 a, u64 b, u64 c) +{ + return vtime_after_eq(a, b) && vtime_before_eq(a, c); +} + +/** + * vtime_in_range_open - Calculate whether a is in the range of [b, c). + * @a: time to test + * @b: beginning of the range + * @c: end of the range + * + * Return: %true is time a is in the range [b, c), otherwise %false. + */ +static inline bool vtime_in_range_open(u64 a, u64 b, u64 c) +{ + return vtime_after_eq(a, b) && vtime_before(a, c); +} + =20 /* * Other helpers --=20 2.47.1 From nobody Wed Dec 17 18:02:52 2025 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 49ADF1C07C0 for ; Mon, 16 Dec 2024 03:12:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734318750; cv=none; b=ExOtYLeqJhLNCahiJ+BAEWrrMt+nrEm074G3YPajtAPM1evYsAKui6jEepr1lQfyQyPFtJv80VqZPtmfXTRLxPblZ5EauQD76pAFjHJFeyaq1Y0D132E481rsNVOyWosDyDeH2QG20hOOAUGZIbcQoapsncYSiIga6ZB6J0MsZs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734318750; c=relaxed/simple; bh=4stYz4/gGsi209Zl3d3WdyhWsyzSZCdqMC7HTz5fhyA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=n4vfSHfPqYsRl6nFAgcpAYC4aSdUmBLkIJoeVPzamYVnIR0zxhypP1W1gJ9JWCCeIihbZGBs6WlxD526/2KNeAyRs4F3JvyqHtiFaZbkEXsKXUWf9Ep+yU8Rbn0V+e+M2gzfji7gnenYymacl6672ESgF0WU3ykmp7JGGliGfn0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=kGYmxhAr; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="kGYmxhAr" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-21145812538so34162705ad.0 for ; Sun, 15 Dec 2024 19:12:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734318748; x=1734923548; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bMZAxlMblUHwQnDYumYsih8SVwxSsCKPzL7XYsgcWJQ=; b=kGYmxhAr69WBCh2qo8O1nXvAiZtRX7C/aUfWcMlwA/SVRclOgBAOGlqG8+mbYxo70L hrKtm5PmV9JJ960ckBg5XOY6NdX+LBClHEvQOUElmgxkEzi0wjKRE+oym9rFhehUnTs1 6MvMvL7PKoQBv/JT4WkySn+HlXjJvM6pTSUqfPmshhTl49kMuwr/wRy8/UtIbBfelqF0 uuzxdEvgw995oskIJFSo3/P5VZDH5V1B9J48YMp3xhwQVXc5IMR047M4IYBzXPGtLTFJ RzubMcopE4rFE2EAT3BKEq9k8O9hDoNO/cjaez92jEMxoDKptK279c6dpKsulXW/rjNj 187A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734318748; x=1734923548; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bMZAxlMblUHwQnDYumYsih8SVwxSsCKPzL7XYsgcWJQ=; b=Ioue4lnc/RHqJ9JhYudCVfm9zL3aKsBrunnjKCBuXM2u6oAgJxGflzqa/DpMCMwIis iHpY3Lk8ziJNAiI8dzXmbqm+jsJFTMuISGZV98iO9+EYYOg8qKjaO5uAkkUOxgQYi4+J SkmsPE8UXeAn0k3ocP0E9yxh8kUYiw9K2cWt4Wf9sWoYy1nZETwYUNsKUm1CTQO2m7QY lLX4mZw3YYOjXdZ0bNEink7y+VAieWn5HymBpJaHXr7TgDztzDdve2LQZOdM5qk2AjlR qco5I4WyXd3l6lE4m4kM3vxUCJOIh571BBMg4DCoU97G4ZnUKslgmgQHukIqA9Z52g9k Mwtg== X-Forwarded-Encrypted: i=1; AJvYcCWaYHLJxPqqZ4hOslSDafpN5rkzeQWT+kTvXoXSsu9rGI4BoFW7BSMrylXZ/tqaZXxlx6z5Z+VQNmvqPno=@vger.kernel.org X-Gm-Message-State: AOJu0YzlqkmtfH9LS/ZrW/vx3A8zHad0ZZlVc0JUJM0rOw+P77ms79tc OiCX6E2HCQF62ir+lsDroxPG0ogyAnvH6dNUhaZMAT/yEgwu0fAk X-Gm-Gg: ASbGncv08cWUx5D78hlfJCUZ4VEkuzV0Nr65imPasIj263exprBuSPKLbE5sOqABCFy lgVQSbgjqt+3Rk80Pb/d9wavmWunX7hHwO2wsvwS9oM06wU4Y0UztmA9yFNPnwyItHXJmuOn3Je JaeVU/wUiTXOQdL7Tj7CxyfpvLiHHEJeh45X9rp+isNaV1Mb7YLDjrO/qGH43qaXrcaxnHs3ZOH jUbQbG88zMnFQB6hZRSK1zteHNx/SvhSOObb6ZNvMxLaZUtONxnHI+sL+aodTnGJXWs7xuwmbkS xg5GIEIjcvyr X-Google-Smtp-Source: AGHT+IE9a/SgVKPHbF5W4uThKayf+TK8X192v/Wi1m6PkR30a4OPCVpl8dIpQQNknshhKk6NDFl2fQ== X-Received: by 2002:a17:903:1d2:b0:216:7cd8:e964 with SMTP id d9443c01a7336-218929e814cmr163850155ad.22.1734318748538; Sun, 15 Dec 2024 19:12:28 -0800 (PST) Received: from localhost ([58.29.143.236]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1e6d0e4sm32874705ad.261.2024.12.15.19.12.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 15 Dec 2024 19:12:28 -0800 (PST) From: Changwoo Min X-Google-Original-From: Changwoo Min To: tj@kernel.org, void@manifault.com, mingo@redhat.com, peterz@infradead.org, arighi@nvidia.com Cc: kernel-dev@igalia.com, linux-kernel@vger.kernel.org, Changwoo Min Subject: [PATCH v5 5/6] sched_ext: Replace bpf_ktime_get_ns() to scx_bpf_now_ns() Date: Mon, 16 Dec 2024 12:11:43 +0900 Message-ID: <20241216031144.98097-6-changwoo@igalia.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241216031144.98097-1-changwoo@igalia.com> References: <20241216031144.98097-1-changwoo@igalia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In the BPF schedulers that use bpf_ktime_get_ns() -- scx_central and scx_flatcg, replace bpf_ktime_get_ns() calls to scx_bpf_now_ns(). Signed-off-by: Changwoo Min --- tools/sched_ext/scx_central.bpf.c | 4 ++-- tools/sched_ext/scx_flatcg.bpf.c | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/tools/sched_ext/scx_central.bpf.c b/tools/sched_ext/scx_centra= l.bpf.c index 2907df78241e..ea1d853b9dd4 100644 --- a/tools/sched_ext/scx_central.bpf.c +++ b/tools/sched_ext/scx_central.bpf.c @@ -245,7 +245,7 @@ void BPF_STRUCT_OPS(central_running, struct task_struct= *p) s32 cpu =3D scx_bpf_task_cpu(p); u64 *started_at =3D ARRAY_ELEM_PTR(cpu_started_at, cpu, nr_cpu_ids); if (started_at) - *started_at =3D bpf_ktime_get_ns() ?: 1; /* 0 indicates idle */ + *started_at =3D scx_bpf_now_ns() ?: 1; /* 0 indicates idle */ } =20 void BPF_STRUCT_OPS(central_stopping, struct task_struct *p, bool runnable) @@ -258,7 +258,7 @@ void BPF_STRUCT_OPS(central_stopping, struct task_struc= t *p, bool runnable) =20 static int central_timerfn(void *map, int *key, struct bpf_timer *timer) { - u64 now =3D bpf_ktime_get_ns(); + u64 now =3D scx_bpf_now_ns(); u64 nr_to_kick =3D nr_queued; s32 i, curr_cpu; =20 diff --git a/tools/sched_ext/scx_flatcg.bpf.c b/tools/sched_ext/scx_flatcg.= bpf.c index 3dbfa82883be..85e33e45f818 100644 --- a/tools/sched_ext/scx_flatcg.bpf.c +++ b/tools/sched_ext/scx_flatcg.bpf.c @@ -734,7 +734,7 @@ void BPF_STRUCT_OPS(fcg_dispatch, s32 cpu, struct task_= struct *prev) struct fcg_cpu_ctx *cpuc; struct fcg_cgrp_ctx *cgc; struct cgroup *cgrp; - u64 now =3D bpf_ktime_get_ns(); + u64 now =3D scx_bpf_now_ns(); bool picked_next =3D false; =20 cpuc =3D find_cpu_ctx(); --=20 2.47.1 From nobody Wed Dec 17 18:02:52 2025 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 754B91C3C1D for ; Mon, 16 Dec 2024 03:12:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734318753; cv=none; b=qWgJP7VeDX5VVH7ZMKyzkooAdWbGpZaAVy9FyI+jKT9XaWsY1ICJrMCek43R+XfCatb9E2L4bqg70EBK1U5u+nqdubaS7wjf78rBVtlo/sWCny5A6OV+G/CSPafYY+fKfub4pthSqnrkid040QNBGURcEk4NI3EDQpLw7ZVArvI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734318753; c=relaxed/simple; bh=QV1F6m7RtxUBxKyIhCZOeu8HRvxTozYvf6p2RqU2fss=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ErgKBZQw5VST1sJK8EXPc12r2I7/ytyramAKVIc/7wk1INwkO5WJNy2nd6Ww8aTkoTOKfjmSa15soNgfoR6Y4NcbnyT9IpYfwR/PIkls39mmmyHcSQV+37UUPSmTlw0j1kq0bAPM/4LMo2LV97BeervwOXxo2bhfsdVWIpnAXhE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=EhRcgSca; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="EhRcgSca" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-216426b0865so30389575ad.0 for ; Sun, 15 Dec 2024 19:12:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734318752; x=1734923552; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rOt0ueFwgFWxNaJC91o0b6u/gfq61/y7Hzh7uJU5b4g=; b=EhRcgScauoUHWxXpBaIbIBXn+b6N9piIvDrKwQBui6cMYFXvNGjQWUSHiC5anBMhQ6 FK3UGW6HHFkVaaazrHsPnGV7wHyXI9/+VUc9CJ6zCV29gYCKYE1Lh5xtHvgs2fN/ZUSf D8UTEBh6QkF3p0Y+Igk7Ko6X40suIEmOSrPPpA5XuwZnI40loU4MS1ZnS/AHkSLwHJwe rxRt632QE+TokJJwFdgSXSZlcx5GN/0shcjzf6b4nQOq4P80RsK9h31KNDO8hX69fbci SFf1R0jnnL5TjEFpXpAup+zsRONQfjG/yDqgipZaUICLwYBZ+V+n7Cxe6loH/tR5/2aJ CFIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734318752; x=1734923552; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rOt0ueFwgFWxNaJC91o0b6u/gfq61/y7Hzh7uJU5b4g=; b=FoowkGbIwkYULrKplT6IOR/shu0FxNb1C1zP1f6ThH04j/s0ooPr1MLyTXmd4XEKuq qPZu2Knvo2ShLPiV/lXsPezn/RRiypD+qzUDPQBcuIcXSSf90fOaxBECLRABPmhx4IZi Oisy9FymQRnqTq4I4zSd9F/h3ZR6dybgAWARICTJqxOE6BjeJHJUFP/ooAMUlPZTDXR+ wPUFg4IBSYerETq3WwXOX5Kf+3S9cV3QwxNCeD5M6uR2Vwnd89qhud8yUNhVwHQbA3nN 4mGf5FAloAfYs4k/uREahNRija/7fJRBZY5bUKCYL28su2vCAUAKp8ieDxbw4uuxPonv pztg== X-Forwarded-Encrypted: i=1; AJvYcCVqWMXMZW59yB1+4t4RI/MEl8aLG0X2rhDgtx7xNHa7Wrw2M/T38bEcOVVygkKLhIYGwK7myyshsOLb0Tg=@vger.kernel.org X-Gm-Message-State: AOJu0YxTp9aBQ75/qN7EPVIY0ypak2pXx2/c23YoXVprgPHaG/f3Vbme Ihgk8jWF0R4K+WDEyqxRcJEb9SlBvbKso4ojpR+q3EPcJGrzPzSF X-Gm-Gg: ASbGncuu3/e0vORGduy4avJmw6m0j0rrFr3GJMUXR7Ktl7FUIHmWMTcQrCukY5EQlcB f0UU2HUhPXA00Ow8pvlQG3lEbYzANqHaA8lc5tr5/K2Lb5Xz1MKUnvZzj6cLummnyaF8MM2VFHO Gxw7wIdX5qx/iWTv570Tcrv9nMMqIlCkYDdZLcP232ZGEZ1GtjgQIn+EayWQDHQKq45jDK+9mOR qoC03AqdTxXgmCavTNn/che18QIXKVIJY+ywmtnxz/kq02Kh9zLNmoUhQZz8lsjoi/ZZyWe/oA5 CDQBKsUnYNbN X-Google-Smtp-Source: AGHT+IH2J1GusDzNvQ+wayqWVdLaBt7wz58A7Yg61gP9ygIEJFz+eBl0003MSwpIbPOi+U0T/4oxcw== X-Received: by 2002:a17:902:f14b:b0:215:b087:5d62 with SMTP id d9443c01a7336-21892a4179bmr118497035ad.36.1734318751757; Sun, 15 Dec 2024 19:12:31 -0800 (PST) Received: from localhost ([58.29.143.236]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1e6d90dsm32918565ad.266.2024.12.15.19.12.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 15 Dec 2024 19:12:31 -0800 (PST) From: Changwoo Min X-Google-Original-From: Changwoo Min To: tj@kernel.org, void@manifault.com, mingo@redhat.com, peterz@infradead.org, arighi@nvidia.com Cc: kernel-dev@igalia.com, linux-kernel@vger.kernel.org, Changwoo Min Subject: [PATCH v5 6/6] sched_ext: Use time helpers in BPF schedulers Date: Mon, 16 Dec 2024 12:11:44 +0900 Message-ID: <20241216031144.98097-7-changwoo@igalia.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241216031144.98097-1-changwoo@igalia.com> References: <20241216031144.98097-1-changwoo@igalia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Modify the BPF schedulers to use time helpers defined in common.bpf.h Signed-off-by: Changwoo Min --- tools/sched_ext/scx_central.bpf.c | 5 ----- tools/sched_ext/scx_flatcg.bpf.c | 11 +++-------- tools/sched_ext/scx_simple.bpf.c | 5 ----- 3 files changed, 3 insertions(+), 18 deletions(-) diff --git a/tools/sched_ext/scx_central.bpf.c b/tools/sched_ext/scx_centra= l.bpf.c index ea1d853b9dd4..c3b6998ea83e 100644 --- a/tools/sched_ext/scx_central.bpf.c +++ b/tools/sched_ext/scx_central.bpf.c @@ -87,11 +87,6 @@ struct { __type(value, struct central_timer); } central_timer SEC(".maps"); =20 -static bool vtime_before(u64 a, u64 b) -{ - return (s64)(a - b) < 0; -} - s32 BPF_STRUCT_OPS(central_select_cpu, struct task_struct *p, s32 prev_cpu, u64 wake_flags) { diff --git a/tools/sched_ext/scx_flatcg.bpf.c b/tools/sched_ext/scx_flatcg.= bpf.c index 85e33e45f818..2735ec25e511 100644 --- a/tools/sched_ext/scx_flatcg.bpf.c +++ b/tools/sched_ext/scx_flatcg.bpf.c @@ -137,11 +137,6 @@ static u64 div_round_up(u64 dividend, u64 divisor) return (dividend + divisor - 1) / divisor; } =20 -static bool vtime_before(u64 a, u64 b) -{ - return (s64)(a - b) < 0; -} - static bool cgv_node_less(struct bpf_rb_node *a, const struct bpf_rb_node = *b) { struct cgv_node *cgc_a, *cgc_b; @@ -920,14 +915,14 @@ void BPF_STRUCT_OPS(fcg_cgroup_move, struct task_stru= ct *p, struct cgroup *from, struct cgroup *to) { struct fcg_cgrp_ctx *from_cgc, *to_cgc; - s64 vtime_delta; + s64 delta; =20 /* find_cgrp_ctx() triggers scx_ops_error() on lookup failures */ if (!(from_cgc =3D find_cgrp_ctx(from)) || !(to_cgc =3D find_cgrp_ctx(to)= )) return; =20 - vtime_delta =3D p->scx.dsq_vtime - from_cgc->tvtime_now; - p->scx.dsq_vtime =3D to_cgc->tvtime_now + vtime_delta; + delta =3D vtime_delta(p->scx.dsq_vtime, from_cgc->tvtime_now); + p->scx.dsq_vtime =3D to_cgc->tvtime_now + delta; } =20 s32 BPF_STRUCT_OPS_SLEEPABLE(fcg_init) diff --git a/tools/sched_ext/scx_simple.bpf.c b/tools/sched_ext/scx_simple.= bpf.c index 31f915b286c6..6561d400ae6d 100644 --- a/tools/sched_ext/scx_simple.bpf.c +++ b/tools/sched_ext/scx_simple.bpf.c @@ -52,11 +52,6 @@ static void stat_inc(u32 idx) (*cnt_p)++; } =20 -static inline bool vtime_before(u64 a, u64 b) -{ - return (s64)(a - b) < 0; -} - s32 BPF_STRUCT_OPS(simple_select_cpu, struct task_struct *p, s32 prev_cpu,= u64 wake_flags) { bool is_idle =3D false; --=20 2.47.1