From nobody Sun May 10 06:27:53 2026 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ABF201A3A8A for ; Mon, 30 Dec 2024 09:56:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735552595; cv=none; b=RYYhZzaGgJEPcpd3u+aYLnMAYDZe+AGKge2HlaRNvpgvPkoeJAtvIWt2KSAEOhUJ9OjygncFgLqGjApt2JBRVU4Fia7FPrl2TPdozOEeJqsmfQy/ATQUzTVN1Ks8skv5rHvVnXmzs5JTcW1cEZsudbRyXzia1AFykH5fWaQb25k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735552595; c=relaxed/simple; bh=TlSqGNDpjyEQ/sAS+WG9j8ZbPrrXCvTJG01tJ8bLoZ0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qUwCnl8Nj/kcvhWxd1N4eXERsiNruOA7Cx+2Ohq9htZeoZDT8MpzLe1zU4ojJNp/hG4nSdQC4/HJlgtol4jevWmtGXztgbi+INxqxf+5P/8ZqVO5g/Vhuqj3YTRndxFyV5zW7Ylm4t9O6EREMmrAVaekFjFmmRmxUYN6+d4PF18= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=A7LG+trg; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="A7LG+trg" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-21675fd60feso154446365ad.2 for ; Mon, 30 Dec 2024 01:56:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1735552593; x=1736157393; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ot9Lx1zg/XLL07layOuusQRDG2mU5e3t2AM5FahhmOs=; b=A7LG+trgk/tlA58cW7+Ei5PTVVGPE6qu27iD0mCAwxFNDXA2pW1kjJcGF8wjuEhpm8 AyEiLoM1dfMJ9ENxcCglpu4oOKkUoux7uLHzijxhO6kVt2z6KY3/qn7uas2NtkNzLGAm ZTIi67rakqWHJtA1qaz3bRwd1nFaZyvU2pONJP5QYT5E8dIx/EL3r/ZKFQ42bTY/2IZd O8Eo8OmWWhbGrFVYLj/9oqBic0chsfJuhfIp8ubXaEOKK0zX5g4VIiJb6jrngDF+Zsih CQFlj/iBUwB6omlJ0F5iROaOPrHvquO/K86fb+BNFvsG5jEQkN4hHq5XXA/rvbe3b7h3 Xi6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735552593; x=1736157393; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ot9Lx1zg/XLL07layOuusQRDG2mU5e3t2AM5FahhmOs=; b=VF3b+TcQFXFKfWkPm1qalYyEwwAanUgANgROsdGGY5EJs7rq23rgBr4pzs4xuEtxhr F4CU/wKBblwWB5NlvTkC/GTglzQdtflAXBfpTUYQ9jFBAbNtAJ4Lmvm3ugXM8IE6yGzT n9vWFfSIvm5xS4gvQ+qj1K+Zgy/yaW7ZOmIYUXe+kYJmDEMcO2d+7abFLVAKjBoQn3/1 nXVbijSkok3Tljgl/e/m9ZEvxvgtRUoTP9ixLfWG1SvYqUl6yCvoobLEIzpRcHub8oYX SpDIyIgSLBvQsjoq9hyL9St+siCUag4/S/LV9lJOKSnyTzmmB2yyO6/iqlepRnymN5ht rw8A== X-Forwarded-Encrypted: i=1; AJvYcCWzXB9HnRu4OwJvUryds4Ncibu/CC0cMB5qqDLqZleTHqXN3FvKxbzVqluUpvRSvogK1UBlzF6SUhM+sAQ=@vger.kernel.org X-Gm-Message-State: AOJu0YxjtgrC6Cp1gy3Lftbwmv2mM2bVc/3pVNqv+Pr2GypAo+waYEAY RdWob/4bJdPCy9024xCaB1gB0TM8rUfVyPCbZwl7Jj/0or5fBDWl0xB60Q== X-Gm-Gg: ASbGncv9D5o1EPdQxN4V+rFd48MoPO+corjw8lis9okAhkX2L0QSbvi8ndqzVcfkpo8 mA5cCTYzh4v5c+HFM/4wDBqnwIwHfmqkJj4txZ5z0icSC4KfsbH+VW45A3e7hLHgUsl+evBsm0X 8dX2EnzK4MhBSJBHC0hxxQXyweAm49NN54gyyCAQX/4cnTAANVllnig30civL4AgmdMSJyxy7Yp +SF76PyMOEjbigNJZmDWFXIyVhI5PYtPbxFoURt5jPCL0ezz0datopwCVdd X-Google-Smtp-Source: AGHT+IFMK4aKJiEcLVhlOIDS2/X1HwBLTO9M/Jq/nvvVRqpxchmaBVGp1bPGkCmPlBzU+BMB9V245w== X-Received: by 2002:a17:903:2449:b0:215:3a42:dc17 with SMTP id d9443c01a7336-219e6e8bbabmr436279965ad.7.1735552593016; Mon, 30 Dec 2024 01:56:33 -0800 (PST) Received: from localhost ([58.29.143.236]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dc9f68e4sm175002415ad.211.2024.12.30.01.56.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:56:32 -0800 (PST) From: Changwoo Min X-Google-Original-From: Changwoo Min To: tj@kernel.org, void@manifault.com, arighi@nvidia.com, mingo@redhat.com, peterz@infradead.org Cc: changwoo@igalia.com, kernel-dev@igalia.com, linux-kernel@vger.kernel.org Subject: [PATCH v7 1/6] sched_ext: Relocate scx_enabled() related code Date: Mon, 30 Dec 2024 18:56:20 +0900 Message-ID: <20241230095625.114363-2-changwoo@igalia.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230095625.114363-1-changwoo@igalia.com> References: <20241230095625.114363-1-changwoo@igalia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" scx_enabled() will be used in scx_rq_clock_update/invalidate() in the following patch, so relocate the scx_enabled() related code to the proper location. Signed-off-by: Changwoo Min Acked-by: Andrea Righi --- kernel/sched/sched.h | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 76f5f53a645f..440ecedf871b 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1717,6 +1717,19 @@ struct rq_flags { =20 extern struct balance_callback balance_push_callback; =20 +#ifdef CONFIG_SCHED_CLASS_EXT +extern const struct sched_class ext_sched_class; + +DECLARE_STATIC_KEY_FALSE(__scx_ops_enabled); /* SCX BPF scheduler loaded */ +DECLARE_STATIC_KEY_FALSE(__scx_switched_all); /* all fair class tasks on S= CX */ + +#define scx_enabled() static_branch_unlikely(&__scx_ops_enabled) +#define scx_switched_all() static_branch_unlikely(&__scx_switched_all) +#else /* !CONFIG_SCHED_CLASS_EXT */ +#define scx_enabled() false +#define scx_switched_all() false +#endif /* !CONFIG_SCHED_CLASS_EXT */ + /* * Lockdep annotation that avoids accidental unlocks; it's like a * sticky/continuous lockdep_assert_held(). @@ -2505,19 +2518,6 @@ extern const struct sched_class rt_sched_class; extern const struct sched_class fair_sched_class; extern const struct sched_class idle_sched_class; =20 -#ifdef CONFIG_SCHED_CLASS_EXT -extern const struct sched_class ext_sched_class; - -DECLARE_STATIC_KEY_FALSE(__scx_ops_enabled); /* SCX BPF scheduler loaded */ -DECLARE_STATIC_KEY_FALSE(__scx_switched_all); /* all fair class tasks on S= CX */ - -#define scx_enabled() static_branch_unlikely(&__scx_ops_enabled) -#define scx_switched_all() static_branch_unlikely(&__scx_switched_all) -#else /* !CONFIG_SCHED_CLASS_EXT */ -#define scx_enabled() false -#define scx_switched_all() false -#endif /* !CONFIG_SCHED_CLASS_EXT */ - /* * Iterate only active classes. SCX can take over all fair tasks or be * completely disabled. If the former, skip fair. If the latter, skip SCX. --=20 2.47.1 From nobody Sun May 10 06:27:53 2026 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B7ACF1A725A for ; Mon, 30 Dec 2024 09:56:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735552598; cv=none; b=nbaYZOg7wGjSRK2uR7YDYQoLQEkd2Ynr82Aj8SjA+vLau1ubb1CKDAnjp4bL03deuxnj+I3U//FnzoDHAO7DpkVi3RvHT7EB4ufjm5sdf6x63JHs66Mugfj1XkZzPXztoR3nu0QrDVsd/C9FrqqtmS9lISPmGdThRj1ffble64o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735552598; c=relaxed/simple; bh=DuKg3GfHjLZy5ATJKKWr3ur8sKYzotbM2Qh+ad25Ub0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=af6wocJrvTppy2yw2Vp/8mvFv3XvfC9p00Um8uOF4W2aDtyKoPeVjXtGavtpnDjHMeklF403Y8ZwCuc4nqh6GJBpZgbSR+WTO1ob/njBTeqZpVOhE4lqktXT3y6ynWYEeKvL90T48WxqMMgwnb0+5mdi57J3p+SzjZXvIXsCpNU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=nUqgTIjo; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nUqgTIjo" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-2161eb95317so119849385ad.1 for ; Mon, 30 Dec 2024 01:56:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1735552596; x=1736157396; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/06QYttfu4ZGL2PBErJGQ15YaDc+46NGVMl2sMfX+LU=; b=nUqgTIjo8UdYR3GwXEU2/c2ZVkq0GJFlZeSXw7J+kKi9YAl7NbX+VeMSYN0QQvrEoI Oil2Wng3k4jOAfhaezs6gPcA+CUErDbTTjX1f/KnaIFVNMyWvkGB364ETZUIOCqK/Xzu OK9pVIDQKPtPb1ZfTYadycYx8MVejsaj8Jt0pKs1LCpSCBVQTD0X7zATOa8ptDF7IgD5 TozpNWn1cpiYuszP94WqdKBWw/lAOspoARpUVhVX54ZZX3aFU7uRqeyC4cWiMA6FfLzv imEL14ywlr1csoWfDfT6nanxGBHJ7oPqHZC29ypzh06bM80ozVz/HQJFIgUmtKYW8Ngo 5O5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735552596; x=1736157396; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/06QYttfu4ZGL2PBErJGQ15YaDc+46NGVMl2sMfX+LU=; b=FX8p57nw/4ROmWp0ffpx/JDYMl7IQpWMjY/DGtfzAXRqWbedbVQaLXWRvhqajWILVD WlI+Ss2BTfG8mshb9nnBFEO4TNN+FoWbpcWCYzwO/Qevk4LhydP88y6oq6E12+NUYMMo t/z0sehFn2HeGafsx7K3U7n8R5klC2VSCJ8iPcn/24Uwr7lDJVs+gyTX/hXF9EGKNxZ6 AgKbMWpsBOOYRQ3SrcLtftmNWdhpQh2RFKSXdh1lw8o8c1EKRqBlCSYcbpZW/kUbON3t /eIJrPipvp6P3SrWSliRCZtiCD0wG8ArW4fMGfvUyszFutaTXriZQF6Q9VNUMgh9vPIM MyLA== X-Forwarded-Encrypted: i=1; AJvYcCWLmIpS6dogrPrPn/BTTy/azbfeUgwIyA7AEuETWUKr7FdKNe+B8UQKKOA1PafUbNqXdWc4OznkABDPsYA=@vger.kernel.org X-Gm-Message-State: AOJu0YyBWF8x8fhDoPe5A1RACtvbykRufVuGWLOoK+pTMlNFWtbF9H3K YxPsU4RCIfRIFCA1iOQz2JZpLDDwpQjPJ/skqF1Lfl7wQVcGHApPWGV89A== X-Gm-Gg: ASbGnct0cq3mfAUSRTjIj1MHA1q+HqELOqrHrtyQw+VBGU9u+YqT4K1O0rxhwFkLRcp KaX311GBdZht2KbOH25kiBB/eQkOa2dd6Lef3YcG7b/g+WWQkbXxuzSo8pbCYeVRE4qn/7DRbYF w7mTCTC4zq44gMeZuobmaDIJkFnHq5Z4ZOfCUFkmJCHMVF/WLW7OsYUSflkWXwENnx1nN8r/8lr lXBSBhwave6IVVC/9O+q72G/huSP3e987cT9r5kbhOSlkGU7XmnJ4X/Rves X-Google-Smtp-Source: AGHT+IFCfX6OXH1wC9aT8ZdcGpSfTFkNQpJKxN29UpB4RIkEujwfnrr+3iOjqZcwML5AlWov5IDTQA== X-Received: by 2002:a05:6a20:2d11:b0:1e1:bd5b:b82a with SMTP id adf61e73a8af0-1e5e081c66amr51034179637.40.1735552595836; Mon, 30 Dec 2024 01:56:35 -0800 (PST) Received: from localhost ([58.29.143.236]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842dc7edbcbsm17272746a12.52.2024.12.30.01.56.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:56:35 -0800 (PST) From: Changwoo Min X-Google-Original-From: Changwoo Min To: tj@kernel.org, void@manifault.com, arighi@nvidia.com, mingo@redhat.com, peterz@infradead.org Cc: changwoo@igalia.com, kernel-dev@igalia.com, linux-kernel@vger.kernel.org Subject: [PATCH v7 2/6] sched_ext: Implement scx_bpf_now() Date: Mon, 30 Dec 2024 18:56:21 +0900 Message-ID: <20241230095625.114363-3-changwoo@igalia.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230095625.114363-1-changwoo@igalia.com> References: <20241230095625.114363-1-changwoo@igalia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Returns a high-performance monotonically non-decreasing clock for the curre= nt CPU. The clock returned is in nanoseconds. It provides the following properties: 1) High performance: Many BPF schedulers call bpf_ktime_get_ns() frequently to account for execution time and track tasks' runtime properties. Unfortunately, in some hardware platforms, bpf_ktime_get_ns() -- which eventually reads a hardware timestamp counter -- is neither performant nor scalable. scx_bpf_now() aims to provide a high-performance clock by using the rq clock in the scheduler core whenever possible. 2) High enough resolution for the BPF scheduler use cases: In most BPF scheduler use cases, the required clock resolution is lower than the most accurate hardware clock (e.g., rdtsc in x86). scx_bpf_now() basically uses the rq clock in the scheduler core whenever it is valid. It considers that the rq clock is valid from the time the rq clock is updated (update_rq_clock) until the rq is unlocked (rq_unpin_lock). 3) Monotonically non-decreasing clock for the same CPU: scx_bpf_now() guarantees the clock never goes backward when comparing them in the same CPU. On the other hand, when comparing clocks in different CPUs, there is no such guarantee -- the clock can go backward. It provides a monotonically *non-decreasing* clock so that it would provide the same clock values in two different scx_bpf_now() calls in the same CPU during the same period of when the rq clock is valid. An rq clock becomes valid when it is updated using update_rq_clock() and invalidated when the rq is unlocked using rq_unpin_lock(). Let's suppose the following timeline in the scheduler core: T1. rq_lock(rq) T2. update_rq_clock(rq) T3. a sched_ext BPF operation T4. rq_unlock(rq) T5. a sched_ext BPF operation T6. rq_lock(rq) T7. update_rq_clock(rq) For [T2, T4), we consider that rq clock is valid (SCX_RQ_CLK_VALID is set), so scx_bpf_now() calls during [T2, T4) (including T3) will return the rq clock updated at T2. For duration [T4, T7), when a BPF scheduler can still call scx_bpf_now() (T5), we consider the rq clock is invalid (SCX_RQ_CLK_VALID is unset at T4). So when calling scx_bpf_now() at T5, we will return a fresh clock value by calling sched_clock_cpu() internally. Also, to prevent getting outdated rq clocks from a previous scx scheduler, invalidate all the rq clocks when unloading a BPF scheduler. One example of calling scx_bpf_now(), when the rq clock is invalid (like T5), is in scx_central [1]. The scx_central scheduler uses a BPF timer for preemptive scheduling. In every msec, the timer callback checks if the currently running tasks exceed their timeslice. At the beginning of the BPF timer callback (central_timerfn in scx_central.bpf.c), scx_central gets the current time. When the BPF timer callback runs, the rq clock could be invalid, the same as T5. In this case, scx_bpf_now() returns a fresh clock value rather than returning the old one (T2). [1] https://github.com/sched-ext/scx/blob/main/scheds/c/scx_central.bpf.c Signed-off-by: Changwoo Min Acked-by: Andrea Righi --- kernel/sched/core.c | 6 +++- kernel/sched/ext.c | 74 +++++++++++++++++++++++++++++++++++++++++++- kernel/sched/sched.h | 25 +++++++++++++-- 3 files changed, 101 insertions(+), 4 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 95e40895a519..ab8015c8cab4 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -789,6 +789,7 @@ static void update_rq_clock_task(struct rq *rq, s64 del= ta) void update_rq_clock(struct rq *rq) { s64 delta; + u64 clock; =20 lockdep_assert_rq_held(rq); =20 @@ -800,11 +801,14 @@ void update_rq_clock(struct rq *rq) SCHED_WARN_ON(rq->clock_update_flags & RQCF_UPDATED); rq->clock_update_flags |=3D RQCF_UPDATED; #endif + clock =3D sched_clock_cpu(cpu_of(rq)); + scx_rq_clock_update(rq, clock); =20 - delta =3D sched_clock_cpu(cpu_of(rq)) - rq->clock; + delta =3D clock - rq->clock; if (delta < 0) return; rq->clock +=3D delta; + update_rq_clock_task(rq, delta); } =20 diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c index 16de619ff2f9..2a830c8a00bd 100644 --- a/kernel/sched/ext.c +++ b/kernel/sched/ext.c @@ -4913,7 +4913,7 @@ static void scx_ops_disable_workfn(struct kthread_wor= k *work) struct task_struct *p; struct rhashtable_iter rht_iter; struct scx_dispatch_q *dsq; - int i, kind; + int i, kind, cpu; =20 kind =3D atomic_read(&scx_exit_kind); while (true) { @@ -4996,6 +4996,15 @@ static void scx_ops_disable_workfn(struct kthread_wo= rk *work) scx_task_iter_stop(&sti); percpu_up_write(&scx_fork_rwsem); =20 + /* + * Invalidate all the rq clocks to prevent getting outdated + * rq clocks from a previous scx scheduler. + */ + for_each_possible_cpu(cpu) { + struct rq *rq =3D cpu_rq(cpu); + scx_rq_clock_invalidate(rq); + } + /* no task is on scx, turn off all the switches and flush in-progress cal= ls */ static_branch_disable(&__scx_ops_enabled); for (i =3D SCX_OPI_BEGIN; i < SCX_OPI_END; i++) @@ -7603,6 +7612,68 @@ __bpf_kfunc struct cgroup *scx_bpf_task_cgroup(struc= t task_struct *p) } #endif =20 +/** + * scx_bpf_now - Returns a high-performance monotonically non-decreasing + * clock for the current CPU. The clock returned is in nanoseconds. + * + * It provides the following properties: + * + * 1) High performance: Many BPF schedulers call bpf_ktime_get_ns() freque= ntly + * to account for execution time and track tasks' runtime properties. + * Unfortunately, in some hardware platforms, bpf_ktime_get_ns() -- which + * eventually reads a hardware timestamp counter -- is neither performant= nor + * scalable. scx_bpf_now() aims to provide a high-performance clock by + * using the rq clock in the scheduler core whenever possible. + * + * 2) High enough resolution for the BPF scheduler use cases: In most BPF + * scheduler use cases, the required clock resolution is lower than the m= ost + * accurate hardware clock (e.g., rdtsc in x86). scx_bpf_now() basically + * uses the rq clock in the scheduler core whenever it is valid. It consi= ders + * that the rq clock is valid from the time the rq clock is updated + * (update_rq_clock) until the rq is unlocked (rq_unpin_lock). + * + * 3) Monotonically non-decreasing clock for the same CPU: scx_bpf_now() + * guarantees the clock never goes backward when comparing them in the sa= me + * CPU. On the other hand, when comparing clocks in different CPUs, there + * is no such guarantee -- the clock can go backward. It provides a + * monotonically *non-decreasing* clock so that it would provide the same + * clock values in two different scx_bpf_now() calls in the same CPU + * during the same period of when the rq clock is valid. + */ +__bpf_kfunc u64 scx_bpf_now(void) +{ + struct rq *rq; + u64 clock; + + preempt_disable(); + + rq =3D this_rq(); + if (READ_ONCE(rq->scx.flags) & SCX_RQ_CLK_VALID) { + /* + * If the rq clock is valid, use the cached rq clock. + * + * Note that scx_bpf_now() is re-entrant between a process + * context and an interrupt context (e.g., timer interrupt). + * However, we don't need to consider the race between them + * because such race is not observable from a caller. + */ + clock =3D READ_ONCE(rq->scx.clock); + } else { + /* + * Otherwise, return a fresh rq clock. + * + * The rq clock is updated outside of the rq lock. + * In this case, keep the updated rq clock invalid so the next + * kfunc call outside the rq lock gets a fresh rq clock. + */ + clock =3D sched_clock_cpu(cpu_of(rq)); + } + + preempt_enable(); + + return clock; +} + __bpf_kfunc_end_defs(); =20 BTF_KFUNCS_START(scx_kfunc_ids_any) @@ -7634,6 +7705,7 @@ BTF_ID_FLAGS(func, scx_bpf_cpu_rq) #ifdef CONFIG_CGROUP_SCHED BTF_ID_FLAGS(func, scx_bpf_task_cgroup, KF_RCU | KF_ACQUIRE) #endif +BTF_ID_FLAGS(func, scx_bpf_now) BTF_KFUNCS_END(scx_kfunc_ids_any) =20 static const struct btf_kfunc_id_set scx_kfunc_set_any =3D { diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 440ecedf871b..e40fdc3e5f22 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -754,6 +754,7 @@ enum scx_rq_flags { SCX_RQ_BAL_PENDING =3D 1 << 2, /* balance hasn't run yet */ SCX_RQ_BAL_KEEP =3D 1 << 3, /* balance decided to keep current */ SCX_RQ_BYPASSING =3D 1 << 4, + SCX_RQ_CLK_VALID =3D 1 << 5, /* RQ clock is fresh and valid */ =20 SCX_RQ_IN_WAKEUP =3D 1 << 16, SCX_RQ_IN_BALANCE =3D 1 << 17, @@ -766,9 +767,10 @@ struct scx_rq { unsigned long ops_qseq; u64 extra_enq_flags; /* see move_task_to_local_dsq() */ u32 nr_running; - u32 flags; u32 cpuperf_target; /* [0, SCHED_CAPACITY_SCALE] */ bool cpu_released; + u32 flags; + u64 clock; /* current per-rq clock -- see scx_bpf_now() */ cpumask_var_t cpus_to_kick; cpumask_var_t cpus_to_kick_if_idle; cpumask_var_t cpus_to_preempt; @@ -1725,9 +1727,28 @@ DECLARE_STATIC_KEY_FALSE(__scx_switched_all); /* all= fair class tasks on SCX */ =20 #define scx_enabled() static_branch_unlikely(&__scx_ops_enabled) #define scx_switched_all() static_branch_unlikely(&__scx_switched_all) + +static inline void scx_rq_clock_update(struct rq *rq, u64 clock) +{ + if (!scx_enabled()) + return; + WRITE_ONCE(rq->scx.clock, clock); + WRITE_ONCE(rq->scx.flags, rq->scx.flags | SCX_RQ_CLK_VALID); +} + +static inline void scx_rq_clock_invalidate(struct rq *rq) +{ + if (!scx_enabled()) + return; + WRITE_ONCE(rq->scx.flags, rq->scx.flags & ~SCX_RQ_CLK_VALID); +} + #else /* !CONFIG_SCHED_CLASS_EXT */ #define scx_enabled() false #define scx_switched_all() false + +static inline void scx_rq_clock_update(struct rq *rq, u64 clock) {} +static inline void scx_rq_clock_invalidate(struct rq *rq) {} #endif /* !CONFIG_SCHED_CLASS_EXT */ =20 /* @@ -1759,7 +1780,7 @@ static inline void rq_unpin_lock(struct rq *rq, struc= t rq_flags *rf) if (rq->clock_update_flags > RQCF_ACT_SKIP) rf->clock_update_flags =3D RQCF_UPDATED; #endif - + scx_rq_clock_invalidate(rq); lockdep_unpin_lock(__rq_lockp(rq), rf->cookie); } =20 --=20 2.47.1 From nobody Sun May 10 06:27:53 2026 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3A4001A76BB for ; Mon, 30 Dec 2024 09:56:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735552600; cv=none; b=YOzRBU1kXPrWYTQ1qgvtPs3DnGn3an9vLsN3UQyR8GldqSNOgHXlNLPqFr35Iwehi8eLM9FWqynujCxpJg79aS3OKv5f8lo9mrUBK2eC+e7VSpZKmgfs0kx+0fY/nqCY2LqMXV4U2LAIpJVw5iPjzlXwyeyAkxqmgYHPBym7Pug= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735552600; c=relaxed/simple; bh=quj8prKTf0T2wGBr3bhLbhhITyZHTAO5tc+XMf0VBZI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Jn//poxqiBAb1osfvUQ4ngWMBpP0Oz0sNmJ9Hg7af/sjOEnObFRqU26ZEoQ5lTVM7ixuFAwBZJ/Qap4mqcOgn+yVmmenglaHLUOnOXgsfCnC+RJQKQJ1hKOQfCbwt4oBBRqK46bxmEhb/40/y//TLsleT6qempTGUCoF2NVTGZY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=f1D4LAu1; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="f1D4LAu1" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-2164b1f05caso114065085ad.3 for ; Mon, 30 Dec 2024 01:56:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1735552598; x=1736157398; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Wow0f2L0ocjtG3ziib/G7eNZl8dh1HZAWkEJfvze4q0=; b=f1D4LAu1UpypRodryKcfdbLKXbt4I53kafFYOnsYfVT4FWu6wnbOoIz6fMapfNYYrX Jb/W0Ug3mUq78m7fhhnR4DtqKNR70t4gpgndZzT5EUnfY2XjRAQ2B/oUpl/b+IkkUiwU JvxEgo84mijDrNcWo0UkLcapotDs+uDJymaZ9Tq+sr29md6sTlLb7zXkTIHY0h2jz7XW Xn4awBkkS3FyEvqi56FhsbhdedY9HaQHpCgqR1Mb8HYhUUCBw+pYrn1UUAE/2pKYdwxV C34wPjgPiIam02TGgKf9i1RGd7x4KyY19XvPOWKKANPrIJ2RcHpID6wLU7qtD08NSqhT hIQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735552598; x=1736157398; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Wow0f2L0ocjtG3ziib/G7eNZl8dh1HZAWkEJfvze4q0=; b=lRMtHTwxZFb8IjmxrpfF2O/yvZYupc/H/aFm8RgFUH3l1V5++o2lqBQGtwfZ4EhOfH W7pxJRtlzUbTJHgLMWsWxdklI+mx5zXa8qvL9qNTbUgfCstWXmdJF3atN4dYVRO7qaq+ 14E8Qj1+ASJkozYRi5jqFwDEaFET5ztDm0nCANV5HLaAw8ltZXjSNy4eTNHFactKIipF EabsCqyu+0XuaXDyNqK+qj7qcqVMMnzKycgnL+eujqVB7fCCPtBLpmY4WeDrrAyf4LN9 VAEq9mScpdjnvopmfyAT6yQ3YDKrZ0SIXLCYnfQtAj1mCq4HaMX0FD6QO49cD8MxcUkh V0IQ== X-Forwarded-Encrypted: i=1; AJvYcCUXMkTAZl0dcR+4db2f9pW3U78HnQklzPabOG8iAaa484mHt9IEb+jtKBqMs23VWaxqsm/jDMhGPptigfs=@vger.kernel.org X-Gm-Message-State: AOJu0YxbtV7oZCmHX18mw3LAyH/8yokk11lTMDcnfzoI+NdOO3XdfD9A inG7/UzMLVxbrlpz670ElISBcRKl5IAqlSuzmZM0R9BQKC68I8AU X-Gm-Gg: ASbGncv0S4kzpoAdSPCQdqbY0j4yXFcQ+mmx1E5lftvTGt/2bfRIJO63G4DVjiMsuSU q0KFHWydb7y9dn0pzLpjVsnE+VfOFmDPBnJPna80yrYvKxoOX58YGh/TRgm6hyrVuphrRSRuy3G Yc3OzqtxLiKQFZhfefQL/NCVZVSYsO7f6qZ9FU2U4pF6UCK9idbW8jcRnp2uWFqVBs/P7FK53Dm vK/3rn1aUCnulujZvwhFn7Jqrmh7vb4ce1ptlfU73byt8e4qO7J7c0nK9ox X-Google-Smtp-Source: AGHT+IFqY3pOmRI9SSTGVIiDXYFXWwXKzCMCLSwiAiS+n1VvJqMrF3Zt+cL2rtzp3FGonqCWoL5jjA== X-Received: by 2002:a17:903:94d:b0:216:46f4:7e30 with SMTP id d9443c01a7336-219e6f11764mr549156605ad.43.1735552598613; Mon, 30 Dec 2024 01:56:38 -0800 (PST) Received: from localhost ([58.29.143.236]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-842b85efb98sm17119554a12.35.2024.12.30.01.56.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:56:38 -0800 (PST) From: Changwoo Min X-Google-Original-From: Changwoo Min To: tj@kernel.org, void@manifault.com, arighi@nvidia.com, mingo@redhat.com, peterz@infradead.org Cc: changwoo@igalia.com, kernel-dev@igalia.com, linux-kernel@vger.kernel.org Subject: [PATCH v7 3/6] sched_ext: Add scx_bpf_now() for BPF scheduler Date: Mon, 30 Dec 2024 18:56:22 +0900 Message-ID: <20241230095625.114363-4-changwoo@igalia.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230095625.114363-1-changwoo@igalia.com> References: <20241230095625.114363-1-changwoo@igalia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" scx_bpf_now() is added to the header files so the BPF scheduler can use it. Signed-off-by: Changwoo Min Acked-by: Andrea Righi --- tools/sched_ext/include/scx/common.bpf.h | 1 + tools/sched_ext/include/scx/compat.bpf.h | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/tools/sched_ext/include/scx/common.bpf.h b/tools/sched_ext/inc= lude/scx/common.bpf.h index 858ba1f438f6..5c9517190713 100644 --- a/tools/sched_ext/include/scx/common.bpf.h +++ b/tools/sched_ext/include/scx/common.bpf.h @@ -76,6 +76,7 @@ bool scx_bpf_task_running(const struct task_struct *p) __= ksym; s32 scx_bpf_task_cpu(const struct task_struct *p) __ksym; struct rq *scx_bpf_cpu_rq(s32 cpu) __ksym; struct cgroup *scx_bpf_task_cgroup(struct task_struct *p) __ksym __weak; +u64 scx_bpf_now(void) __ksym __weak; =20 /* * Use the following as @it__iter when calling scx_bpf_dsq_move[_vtime]() = from diff --git a/tools/sched_ext/include/scx/compat.bpf.h b/tools/sched_ext/inc= lude/scx/compat.bpf.h index d56520100a26..50e1499ae093 100644 --- a/tools/sched_ext/include/scx/compat.bpf.h +++ b/tools/sched_ext/include/scx/compat.bpf.h @@ -125,6 +125,11 @@ bool scx_bpf_dispatch_vtime_from_dsq___compat(struct b= pf_iter_scx_dsq *it__iter, false; \ }) =20 +#define scx_bpf_now() \ + (bpf_ksym_exists(scx_bpf_now) ? \ + scx_bpf_now() : \ + bpf_ktime_get_ns()) + /* * Define sched_ext_ops. This may be expanded to define multiple variants = for * backward compatibility. See compat.h::SCX_OPS_LOAD/ATTACH(). --=20 2.47.1 From nobody Sun May 10 06:27:53 2026 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B40091A3BD8 for ; Mon, 30 Dec 2024 09:56:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735552603; cv=none; b=J9OHUSNhi7Z38vprB8Wjd9GVvCTKKgV7Mxr2lBBYWy5kkJHhWNtTxtQw3H/JlazUMSl4B4xjbMJYT72gtmenVmfUmREVy11hLAeoaz1Z8m1E5ZhG3KAnNTm9OvKppubGtl59n1m6mrsP/na9h378LnB4Y/EENH/o3pU5fl2ZGxY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735552603; c=relaxed/simple; bh=vr8H5CtZ05R7JvmYMoz5R6sWD6J+X4CMRtIVG4kOE2o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VM6CNz5ZMO9k2cAPhmJZoj+DAme9ZZd5DgGvyiSOVa4uPLonR43HR7qivLjzh2cQwaTym1w16NpBXfv+LNNricsDZOCkrTRIiekE/Bzn5iXNW6FMbNWkDaSZkLCBe8aD2JP4W+M0VfPoUR40WxCLXakMEpyRLTQAThGfKcFPsxU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=YOPN7GMs; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YOPN7GMs" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-2166f1e589cso140755075ad.3 for ; Mon, 30 Dec 2024 01:56:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1735552601; x=1736157401; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=aQh2aQFFIOtlO17ZAyGjilQkwjee2R8E6fn9IlabB6s=; b=YOPN7GMsG/aBczA1aUZLSoSrVRUibdEncKxy6vVzlFyxH2Ej6M2w+XstPFEYbvRIYM fwfQ3WYPb0FcUi1OmRPzBB4mrro7O+Bs8gUSn6AxQYuscUvX87QhmIjnRZP+NB8kjgZ1 zmQ6bS1LXcmNN2R5JBRbCz2LWslJTngACxxTg3jH1B6hH24XaAUsTwMEskdSVt5bBfWV uGvRoS5Ao314YvPDA/mnkeFTWPyLmmNTuIkBTzMmdpbyhPba4r85wdFsXCIB91oSheLJ xWLP9RutcdJgD3lcUisOzOIA/jXKV5OeTWQLclEqSkBs48u1FymvkSr80/plfiMQPbwc HorQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735552601; x=1736157401; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aQh2aQFFIOtlO17ZAyGjilQkwjee2R8E6fn9IlabB6s=; b=Xj+ys4d5XiXADdzxWy9q2aIdfRu4yWvGkqbGJh2id554DA49AQkKLNnLIPjjxwXC80 Lat80YxZZx+JAMZU3jHV5YSkJZF3zZ7vK8DbKI2gVMFJXxIBr05dWzRQZMgLnl6mIeeg 8tzKLlhFDTFe/ke9QDm8hKdsTAuXG7FcMBgzBp2wyV0WgY3k7WZROEUj3Eo5SylkD0Xo CMzppbMC5x3dtttcrsz9X7CABwHemTho9guHwD4dTtf+w5f+/YRlMqSKZmuFdEPPPX3D vwP/7wA/Gqef1ApuFPZqSBFU+0Npq8y/HJtM45R89IRAjoKNFCJ1SeSRyshw4xeok+ZF 9tIQ== X-Forwarded-Encrypted: i=1; AJvYcCX389+cbXepEFOoPTeg6dSE5gvbKoGtkeWAIgzQfVXr8QwduwtgVyg3qQMa/3GbQaR3/cp7SwllYuqWjHk=@vger.kernel.org X-Gm-Message-State: AOJu0YwHa56/giVOcMO58YWKPinCrlMLSJTFYhgbU9WZl7i32P9CnGJA q6SePC+0ZH4WF24qeBqyTBqX8gfEMpE06GvaTE80/vjB4LSAtent X-Gm-Gg: ASbGnctxyYvqLw3RedgcyXUe+Sfh0gTnqv2hrfoy0OZ7FB4vH19tMkVBldxjp+K8vr3 6v9lIo2dmoloGr3Ak2AXh+ipHbkQ2Nth8wSkMfIOJ4L2zNzrL4KVragitI+0mXuHGcygRKhvEOq ICXqw8YANx9iE19nYRjtzvggKdLWNwvsq3O8i2tCjBmHvi/SO4zP+mycrIF9t1MhPYm/Prw5m7R 8aT6vDzqaY4h84p1R0jdk1pf7+GAVFH/IyUWZrHQGkTtZfoIi8ED/TOLTYx X-Google-Smtp-Source: AGHT+IEmXydpRV/lgFEredU1uGyL/9po+HtJ1WRnt1KYdF7XgXNUkjqWypbSmCRwpW7YEmtg5XbphA== X-Received: by 2002:a05:6a21:8cc2:b0:1e1:f281:8d36 with SMTP id adf61e73a8af0-1e5e0458dbdmr51666350637.10.1735552601046; Mon, 30 Dec 2024 01:56:41 -0800 (PST) Received: from localhost ([58.29.143.236]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-882ec09606csm12720676a12.14.2024.12.30.01.56.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:56:40 -0800 (PST) From: Changwoo Min X-Google-Original-From: Changwoo Min To: tj@kernel.org, void@manifault.com, arighi@nvidia.com, mingo@redhat.com, peterz@infradead.org Cc: changwoo@igalia.com, kernel-dev@igalia.com, linux-kernel@vger.kernel.org Subject: [PATCH v7 4/6] sched_ext: Add time helpers for BPF schedulers Date: Mon, 30 Dec 2024 18:56:23 +0900 Message-ID: <20241230095625.114363-5-changwoo@igalia.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230095625.114363-1-changwoo@igalia.com> References: <20241230095625.114363-1-changwoo@igalia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The following functions are added for BPF schedulers: - time_delta(after, before) - time_after(a, b) - time_before(a, b) - time_after_eq(a, b) - time_before_eq(a, b) - time_in_range(a, b, c) - time_in_range_open(a, b, c) Signed-off-by: Changwoo Min Acked-by: Andrea Righi --- tools/sched_ext/include/scx/common.bpf.h | 94 ++++++++++++++++++++++++ 1 file changed, 94 insertions(+) diff --git a/tools/sched_ext/include/scx/common.bpf.h b/tools/sched_ext/inc= lude/scx/common.bpf.h index 5c9517190713..f3e15e9efa76 100644 --- a/tools/sched_ext/include/scx/common.bpf.h +++ b/tools/sched_ext/include/scx/common.bpf.h @@ -408,6 +408,100 @@ static __always_inline const struct cpumask *cast_mas= k(struct bpf_cpumask *mask) void bpf_rcu_read_lock(void) __ksym; void bpf_rcu_read_unlock(void) __ksym; =20 +/* + * Time helpers, most of which are from jiffies.h. + */ + +/** + * time_delta - Calculate the delta between new and old time stamp + * @after: first comparable as u64 + * @before: second comparable as u64 + * + * Return: the time difference, which is >=3D 0 + */ +static inline s64 time_delta(u64 after, u64 before) +{ + return (s64)(after - before) > 0 ? : 0; +} + +/** + * time_after - returns true if the time a is after time b. + * @a: first comparable as u64 + * @b: second comparable as u64 + * + * Do this with "<0" and ">=3D0" to only test the sign of the result. A + * good compiler would generate better code (and a really good compiler + * wouldn't care). Gcc is currently neither. + * + * Return: %true is time a is after time b, otherwise %false. + */ +static inline bool time_after(u64 a, u64 b) +{ + return (s64)(b - a) < 0; +} + +/** + * time_before - returns true if the time a is before time b. + * @a: first comparable as u64 + * @b: second comparable as u64 + * + * Return: %true is time a is before time b, otherwise %false. + */ +static inline bool time_before(u64 a, u64 b) +{ + return time_after(b, a); +} + +/** + * time_after_eq - returns true if the time a is after or the same as time= b. + * @a: first comparable as u64 + * @b: second comparable as u64 + * + * Return: %true is time a is after or the same as time b, otherwise %fals= e. + */ +static inline bool time_after_eq(u64 a, u64 b) +{ + return (s64)(a - b) >=3D 0; +} + +/** + * time_before_eq - returns true if the time a is before or the same as ti= me b. + * @a: first comparable as u64 + * @b: second comparable as u64 + * + * Return: %true is time a is before or the same as time b, otherwise %fal= se. + */ +static inline bool time_before_eq(u64 a, u64 b) +{ + return time_after_eq(b, a); +} + +/** + * time_in_range - Calculate whether a is in the range of [b, c]. + * @a: time to test + * @b: beginning of the range + * @c: end of the range + * + * Return: %true is time a is in the range [b, c], otherwise %false. + */ +static inline bool time_in_range(u64 a, u64 b, u64 c) +{ + return time_after_eq(a, b) && time_before_eq(a, c); +} + +/** + * time_in_range_open - Calculate whether a is in the range of [b, c). + * @a: time to test + * @b: beginning of the range + * @c: end of the range + * + * Return: %true is time a is in the range [b, c), otherwise %false. + */ +static inline bool time_in_range_open(u64 a, u64 b, u64 c) +{ + return time_after_eq(a, b) && time_before(a, c); +} + =20 /* * Other helpers --=20 2.47.1 From nobody Sun May 10 06:27:53 2026 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5F47D1A4B69 for ; Mon, 30 Dec 2024 09:56:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735552605; cv=none; b=nPkhl7BGWauhnYAWLXQbrCYQTBROAEWQZizu/7RORe8MFSZlc9px6T+GaLg1Df5Mgm9f+aZOzyHQveYUKJUgrzTSVXCz7xc/pg8JiuRjQsw133I4BE31MR0bKcNdFt3on1Uihu3F3y7CDWUknKnbHo70kbY1x7AIz8bB4vi+0Ik= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735552605; c=relaxed/simple; bh=3KWIG2mmKhaV/gJkcgP4kuB7LjMEjgZ5X9Pyt9i8Jhg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bQ9/eGojMpEU4RBi4sUM4gBjrTQOBMxKal1NPuh2BB9hRqNctX5+zNGRqJ3yINANvqRI5GuW7Vg7xHliJrE4PrC9iTZMwHb16EjM8lvOUsZuSFsjdjE63x6838/EPAGCGsaVWCi0bhkRvyiJaSTv/ixFqrXJ3s8BN1AFSkWiZBc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=mB/8PSR5; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mB/8PSR5" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-216634dd574so77422655ad.2 for ; Mon, 30 Dec 2024 01:56:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1735552604; x=1736157404; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=l78V4ESiWJCLASUcLE3njLXpFe+ymfOJGKR+C9fLFbw=; b=mB/8PSR552YRT6oWn/v7bGpc3l2EhjxheYkRW2lVpBiwm/p/G72w3qID7c6RWinnUz NJUc1Jb+alXVGJopuqa4D7DUGwKyDgABucjbs3rSXPIjjdQy6xgOU10jBVtLIXYcnP44 97gz2sfmR69i2RtCNypfONhEQTvu0JrK0b1i2xPoSqeM1KvwS14A7832NxkUZenuriUb OkRrGEc+z/ecAsjU10jWJlw8CaFPR8ve++hpmLFJS1q6xKFVWkYUdhnrYBQrpGJ5ixpM aOQA++O7W4MsdPaMCf555llbL0lZ6XDimraSUB7iczKwC2lpX+/RxJUJKkWWUQ8+hgV4 6Y7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735552604; x=1736157404; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=l78V4ESiWJCLASUcLE3njLXpFe+ymfOJGKR+C9fLFbw=; b=Sr4yV5CNrm1PXcP3VOr5RiyzGCxK80XGDEBOboMc6nHVdAg2teX1/6hTKcn76ok3Yd AG5IxshqbD6EzIWUwcG7gXTUfJZtEKouICwkMeqep47OrSem9N1SLMStDTalPP9CCj2U QMeMAJkX/l3zlBZqnCqhdAUW2PYwV+yqPlWAh/O0BZOdVkXLJ8DdVKOv/2U/dQIDElOR xnMj6V7aw/VbsYGuiQOD073eZbjLTgd1R67Q+9QgWfj/Y+9OqwSbKFhuFM+cgPPL31Wk tCvbPwUFer+JR7kjwlaDJCQ6NVjdIuXup1w453wb5gXnQ4i6oq1GRmTcddeaYT9MD+04 V8qg== X-Forwarded-Encrypted: i=1; AJvYcCW3pPZv1w4WSov2kLiZdtlAsp6jYx7SdF0+qUBRpBTxRUTiLi9CapK4Rk66ayFpLlOiLN7MLuajvl/WUQs=@vger.kernel.org X-Gm-Message-State: AOJu0YysZj5xaykyn7aOSyolI/8kJ1GIJ4seGn6bThklbfsJNSzXHfVg e2/4sP3s7rTzC3dmlHphlcqolVZVU0PQEDucD2mr+WdVIY9o3moL X-Gm-Gg: ASbGncvZqxWMCUnJ2ZMOXfwrla0AYuEaKX9cAesLS8dsUe1uDXxhEwbiV6nlFdrSfbm goFrmymgZT5+PoCEg0Bt+SdfYFwLhOaDXUH5tietpqYLKJK0ZHpQne3XQcjixyOh5OTK2taS59M duPSDuZ3B/SSBE2FaytLWtL/p3Q/s8gMp8tU9q5t0ToH4JxyrhrShZuaKmXO7Ax1HlTpZdjPOCU NrKDv4Nds9edbT7+8MkFTe2YxU/50k9UXkYd9tD6IDvMqsYcYysF/z3p3Df X-Google-Smtp-Source: AGHT+IH/9SlrS19hUpGnSf8nyxdeZpzeFP3jUPMiGiLXZnnBhH9eTvEd3+e2Ad7cF77X2gJaxHO8vQ== X-Received: by 2002:a05:6a20:c705:b0:1e1:9fef:e96a with SMTP id adf61e73a8af0-1e5e044b029mr50503436637.6.1735552603667; Mon, 30 Dec 2024 01:56:43 -0800 (PST) Received: from localhost ([58.29.143.236]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-8e2478b59bcsm5262037a12.72.2024.12.30.01.56.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:56:43 -0800 (PST) From: Changwoo Min X-Google-Original-From: Changwoo Min To: tj@kernel.org, void@manifault.com, arighi@nvidia.com, mingo@redhat.com, peterz@infradead.org Cc: changwoo@igalia.com, kernel-dev@igalia.com, linux-kernel@vger.kernel.org Subject: [PATCH v7 5/6] sched_ext: Replace bpf_ktime_get_ns() to scx_bpf_now() Date: Mon, 30 Dec 2024 18:56:24 +0900 Message-ID: <20241230095625.114363-6-changwoo@igalia.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230095625.114363-1-changwoo@igalia.com> References: <20241230095625.114363-1-changwoo@igalia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In the BPF schedulers that use bpf_ktime_get_ns() -- scx_central and scx_flatcg, replace bpf_ktime_get_ns() calls to scx_bpf_now(). Signed-off-by: Changwoo Min Acked-by: Andrea Righi --- tools/sched_ext/scx_central.bpf.c | 4 ++-- tools/sched_ext/scx_flatcg.bpf.c | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/tools/sched_ext/scx_central.bpf.c b/tools/sched_ext/scx_centra= l.bpf.c index 2907df78241e..4239034ad593 100644 --- a/tools/sched_ext/scx_central.bpf.c +++ b/tools/sched_ext/scx_central.bpf.c @@ -245,7 +245,7 @@ void BPF_STRUCT_OPS(central_running, struct task_struct= *p) s32 cpu =3D scx_bpf_task_cpu(p); u64 *started_at =3D ARRAY_ELEM_PTR(cpu_started_at, cpu, nr_cpu_ids); if (started_at) - *started_at =3D bpf_ktime_get_ns() ?: 1; /* 0 indicates idle */ + *started_at =3D scx_bpf_now() ?: 1; /* 0 indicates idle */ } =20 void BPF_STRUCT_OPS(central_stopping, struct task_struct *p, bool runnable) @@ -258,7 +258,7 @@ void BPF_STRUCT_OPS(central_stopping, struct task_struc= t *p, bool runnable) =20 static int central_timerfn(void *map, int *key, struct bpf_timer *timer) { - u64 now =3D bpf_ktime_get_ns(); + u64 now =3D scx_bpf_now(); u64 nr_to_kick =3D nr_queued; s32 i, curr_cpu; =20 diff --git a/tools/sched_ext/scx_flatcg.bpf.c b/tools/sched_ext/scx_flatcg.= bpf.c index 3dbfa82883be..5f588963fb2f 100644 --- a/tools/sched_ext/scx_flatcg.bpf.c +++ b/tools/sched_ext/scx_flatcg.bpf.c @@ -734,7 +734,7 @@ void BPF_STRUCT_OPS(fcg_dispatch, s32 cpu, struct task_= struct *prev) struct fcg_cpu_ctx *cpuc; struct fcg_cgrp_ctx *cgc; struct cgroup *cgrp; - u64 now =3D bpf_ktime_get_ns(); + u64 now =3D scx_bpf_now(); bool picked_next =3D false; =20 cpuc =3D find_cpu_ctx(); --=20 2.47.1 From nobody Sun May 10 06:27:53 2026 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 951AF1A4F09 for ; Mon, 30 Dec 2024 09:56:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735552608; cv=none; b=mUdbkoqVDagacUU+fji3i00crhFNxXW0iXDHCqkqP/gUziXcG6U7Axmvxce8OTpNNz24FyY/gm67kYny7Lv+06/SwGDH98Z+BTfzQsoM2Vanja8zXSlVUkvoTxy/SaHf1gS6JIVkRu8PtTmywhBhHDSMyzewf90Cv18EyVOyMaE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735552608; c=relaxed/simple; bh=x8H6DVoIMlNDpp4A25ArxzyiMF448S9aPxeDVLI8SLU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CJ2uLmgUD1yTPb6CqJMnffDkKxHVZoEcUDMb7Je+qtzNYAmLmN6O4vps7D/kkogUH27f7k0ySA/whTW0z+IoriTVG9TQr/V6DlZOvcloEGS8vFQl0giBKAK0ew2Urkpr4nWMqJomA3ydP/HqQNQYEBD2zaliHSk7OEmzn+lXlYY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=PmlW4jID; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="PmlW4jID" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-2161eb95317so119850725ad.1 for ; Mon, 30 Dec 2024 01:56:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1735552606; x=1736157406; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rmZA4vQufn+h6OnUPeuDpOx8Poi6ZPcPUwnKJ0yEQjk=; b=PmlW4jIDTNB8sYC27OBJwt1M+E2jq0N9vA4ymI7CFusKB6V3eZ3Tdfe/jJdc3zRcjy jSVMz53f7wy0GtLQAclGOoeQoZ5POCaOZixfr0TGxe1UnZlpTHFTZjH2TSAsRS7cCOhq 1Kg5A9dY9BZIoXbv7FwGHjeuLV0upqLGuhbBoLStPFIc6jrMtXay5R6CrfPYneggPJ5T QJpaLZgMcJbM92SDHSNgckJRXTqcXjco6enYTlC/j7XWu9qqUjogNg1jfJsGeLv/KJow AxbS64yKfcVxG0WjbJp9JTSuKt32fFIcCNYEBiNi8twQtx9Wxv8vfiQUT+Jsvp2nSBem 9HTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735552606; x=1736157406; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rmZA4vQufn+h6OnUPeuDpOx8Poi6ZPcPUwnKJ0yEQjk=; b=L0GXmYSW6rRAt4v3Lgl2140OY6IDqMcMF2DKQk+yodxjVqM5ylqOa6yomxGkGP3jiP xuuksbLfgdarqeKT5f1gfaDaMg0Cg7JRgd07jlNhAXvxgpNU4qPHAhrnd1Zbkbysia3t veOXwdXyHJK/EOEfD+v2jf4Y98TXYWIl8ZoNan++Bz35nZfed+bGl+fwGNrkqvGg2jpT CbZ3N8N/OvcfbiaQg3xVtA7lXLCDGMPuCgQry20u0zrueapk0EUYmPmWqYzNxCbBwp3W 9OGVtlHL9ddVm4iy88Jbu9CDveFRafmqQVg7bAhwLGhuKZMRFX3YrHG1jpuENPD333pq BDfw== X-Forwarded-Encrypted: i=1; AJvYcCUFdGHdEIEFgokvvc51H9sNhJsF9IPSjvLNUlpu2+glXd4D8xgNX4X23Lwb36knhZr9OLnocjQN0r0WHic=@vger.kernel.org X-Gm-Message-State: AOJu0YwvDx85S6HK7DiPawaXuUo2/wJvh48zFZdTkbjgFoYrh2GrqD5o x9hzyjml6f7j5xpA3NSKb+kKETKVnU0vlB1Zitb1cVX3yoFkDF677qHEXw== X-Gm-Gg: ASbGncvoFyRKLhlovr+elyE8ebo/OLYcn1iZvgsV69PBe+1Z4Z/OsRSv0DDmvk8ygWJ sYCcVHUDCiTi5oVKYnTppBuurOtH9zhO1MCJTJ3VMkcc7HfwPjsCDoVjdZ3mgvVuIBGR0cPiVNm xSL1fKDOj4QblP6dZaC5v1ioJnZyWyK3xULnho74qosCD0Ckg++DZJidtnicwISu3kDyjfnkey5 fRxvmpbiriycahCA0xIr59AC67qtryStII1C8yrl24lI/urbKTkF572NHCi X-Google-Smtp-Source: AGHT+IG6q4klahbK+NORht++i7GfPZ9QazKnUpxFF1PsCwFxgtUJlZpFhvYeDljqSfvjWw7SMSoy5w== X-Received: by 2002:a17:903:2310:b0:215:e685:fa25 with SMTP id d9443c01a7336-219e6e9fa7fmr438056645ad.20.1735552605834; Mon, 30 Dec 2024 01:56:45 -0800 (PST) Received: from localhost ([58.29.143.236]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dc972b34sm175991835ad.81.2024.12.30.01.56.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Dec 2024 01:56:45 -0800 (PST) From: Changwoo Min X-Google-Original-From: Changwoo Min To: tj@kernel.org, void@manifault.com, arighi@nvidia.com, mingo@redhat.com, peterz@infradead.org Cc: changwoo@igalia.com, kernel-dev@igalia.com, linux-kernel@vger.kernel.org Subject: [PATCH v7 6/6] sched_ext: Use time helpers in BPF schedulers Date: Mon, 30 Dec 2024 18:56:25 +0900 Message-ID: <20241230095625.114363-7-changwoo@igalia.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230095625.114363-1-changwoo@igalia.com> References: <20241230095625.114363-1-changwoo@igalia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Modify the BPF schedulers to use time helpers defined in common.bpf.h Signed-off-by: Changwoo Min Acked-by: Andrea Righi --- tools/sched_ext/scx_central.bpf.c | 7 +------ tools/sched_ext/scx_flatcg.bpf.c | 21 ++++++++------------- tools/sched_ext/scx_simple.bpf.c | 9 ++------- 3 files changed, 11 insertions(+), 26 deletions(-) diff --git a/tools/sched_ext/scx_central.bpf.c b/tools/sched_ext/scx_centra= l.bpf.c index 4239034ad593..50bc1737c167 100644 --- a/tools/sched_ext/scx_central.bpf.c +++ b/tools/sched_ext/scx_central.bpf.c @@ -87,11 +87,6 @@ struct { __type(value, struct central_timer); } central_timer SEC(".maps"); =20 -static bool vtime_before(u64 a, u64 b) -{ - return (s64)(a - b) < 0; -} - s32 BPF_STRUCT_OPS(central_select_cpu, struct task_struct *p, s32 prev_cpu, u64 wake_flags) { @@ -279,7 +274,7 @@ static int central_timerfn(void *map, int *key, struct = bpf_timer *timer) /* kick iff the current one exhausted its slice */ started_at =3D ARRAY_ELEM_PTR(cpu_started_at, cpu, nr_cpu_ids); if (started_at && *started_at && - vtime_before(now, *started_at + slice_ns)) + time_before(now, *started_at + slice_ns)) continue; =20 /* and there's something pending */ diff --git a/tools/sched_ext/scx_flatcg.bpf.c b/tools/sched_ext/scx_flatcg.= bpf.c index 5f588963fb2f..2c720e3ecad5 100644 --- a/tools/sched_ext/scx_flatcg.bpf.c +++ b/tools/sched_ext/scx_flatcg.bpf.c @@ -137,11 +137,6 @@ static u64 div_round_up(u64 dividend, u64 divisor) return (dividend + divisor - 1) / divisor; } =20 -static bool vtime_before(u64 a, u64 b) -{ - return (s64)(a - b) < 0; -} - static bool cgv_node_less(struct bpf_rb_node *a, const struct bpf_rb_node = *b) { struct cgv_node *cgc_a, *cgc_b; @@ -271,7 +266,7 @@ static void cgrp_cap_budget(struct cgv_node *cgv_node, = struct fcg_cgrp_ctx *cgc) */ max_budget =3D (cgrp_slice_ns * nr_cpus * cgc->hweight) / (2 * FCG_HWEIGHT_ONE); - if (vtime_before(cvtime, cvtime_now - max_budget)) + if (time_before(cvtime, cvtime_now - max_budget)) cvtime =3D cvtime_now - max_budget; =20 cgv_node->cvtime =3D cvtime; @@ -401,7 +396,7 @@ void BPF_STRUCT_OPS(fcg_enqueue, struct task_struct *p,= u64 enq_flags) * Limit the amount of budget that an idling task can accumulate * to one slice. */ - if (vtime_before(tvtime, cgc->tvtime_now - SCX_SLICE_DFL)) + if (time_before(tvtime, cgc->tvtime_now - SCX_SLICE_DFL)) tvtime =3D cgc->tvtime_now - SCX_SLICE_DFL; =20 scx_bpf_dsq_insert_vtime(p, cgrp->kn->id, SCX_SLICE_DFL, @@ -535,7 +530,7 @@ void BPF_STRUCT_OPS(fcg_running, struct task_struct *p) * from multiple CPUs and thus racy. Any error should be * contained and temporary. Let's just live with it. */ - if (vtime_before(cgc->tvtime_now, p->scx.dsq_vtime)) + if (time_before(cgc->tvtime_now, p->scx.dsq_vtime)) cgc->tvtime_now =3D p->scx.dsq_vtime; } bpf_cgroup_release(cgrp); @@ -645,7 +640,7 @@ static bool try_pick_next_cgroup(u64 *cgidp) cgv_node =3D container_of(rb_node, struct cgv_node, rb_node); cgid =3D cgv_node->cgid; =20 - if (vtime_before(cvtime_now, cgv_node->cvtime)) + if (time_before(cvtime_now, cgv_node->cvtime)) cvtime_now =3D cgv_node->cvtime; =20 /* @@ -744,7 +739,7 @@ void BPF_STRUCT_OPS(fcg_dispatch, s32 cpu, struct task_= struct *prev) if (!cpuc->cur_cgid) goto pick_next_cgroup; =20 - if (vtime_before(now, cpuc->cur_at + cgrp_slice_ns)) { + if (time_before(now, cpuc->cur_at + cgrp_slice_ns)) { if (scx_bpf_dsq_move_to_local(cpuc->cur_cgid)) { stat_inc(FCG_STAT_CNS_KEEP); return; @@ -920,14 +915,14 @@ void BPF_STRUCT_OPS(fcg_cgroup_move, struct task_stru= ct *p, struct cgroup *from, struct cgroup *to) { struct fcg_cgrp_ctx *from_cgc, *to_cgc; - s64 vtime_delta; + s64 delta; =20 /* find_cgrp_ctx() triggers scx_ops_error() on lookup failures */ if (!(from_cgc =3D find_cgrp_ctx(from)) || !(to_cgc =3D find_cgrp_ctx(to)= )) return; =20 - vtime_delta =3D p->scx.dsq_vtime - from_cgc->tvtime_now; - p->scx.dsq_vtime =3D to_cgc->tvtime_now + vtime_delta; + delta =3D time_delta(p->scx.dsq_vtime, from_cgc->tvtime_now); + p->scx.dsq_vtime =3D to_cgc->tvtime_now + delta; } =20 s32 BPF_STRUCT_OPS_SLEEPABLE(fcg_init) diff --git a/tools/sched_ext/scx_simple.bpf.c b/tools/sched_ext/scx_simple.= bpf.c index 31f915b286c6..e6de99dba7db 100644 --- a/tools/sched_ext/scx_simple.bpf.c +++ b/tools/sched_ext/scx_simple.bpf.c @@ -52,11 +52,6 @@ static void stat_inc(u32 idx) (*cnt_p)++; } =20 -static inline bool vtime_before(u64 a, u64 b) -{ - return (s64)(a - b) < 0; -} - s32 BPF_STRUCT_OPS(simple_select_cpu, struct task_struct *p, s32 prev_cpu,= u64 wake_flags) { bool is_idle =3D false; @@ -84,7 +79,7 @@ void BPF_STRUCT_OPS(simple_enqueue, struct task_struct *p= , u64 enq_flags) * Limit the amount of budget that an idling task can accumulate * to one slice. */ - if (vtime_before(vtime, vtime_now - SCX_SLICE_DFL)) + if (time_before(vtime, vtime_now - SCX_SLICE_DFL)) vtime =3D vtime_now - SCX_SLICE_DFL; =20 scx_bpf_dsq_insert_vtime(p, SHARED_DSQ, SCX_SLICE_DFL, vtime, @@ -108,7 +103,7 @@ void BPF_STRUCT_OPS(simple_running, struct task_struct = *p) * thus racy. Any error should be contained and temporary. Let's just * live with it. */ - if (vtime_before(vtime_now, p->scx.dsq_vtime)) + if (time_before(vtime_now, p->scx.dsq_vtime)) vtime_now =3D p->scx.dsq_vtime; } =20 --=20 2.47.1