From nobody Thu Dec 18 09:42:49 2025 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9E2805FEED for ; Wed, 18 Dec 2024 08:02:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734508936; cv=none; b=G9EsV/rhcMb+J0jMoXNeNzubEOXx2huWm9D9cBtJm3uYQPf0KcXP1LOYiFkpOVkLM8WjMmxZBVBIw9/uuuzZoNVmtRFO9GNv/QIGBmIrUhSSkneoMGsEM+yXwpvie4M2/lTpPiqEn9ver4EFak9Dtmpf4TGa+EMs3JM7TnhxTQQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734508936; c=relaxed/simple; bh=+U6M4TUc+OXtx5WOq6rwzbt8b8OsFAzVNWZHTUOM2f4=; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version; b=rYUxAP8eFQHzjQgs1NyYQaXdj8LO8teo4VDBMYHnn4h9rv9OHCjquqZZbscKS7f5BPEQ/xV4OlHTd9rjAjvR0oPsmTVx4zCzzibJ88tfpDhSvYzrAekSNXZh8DpiI4ZXuX4lrL4hcGT2H0/WKv0NQ546+peO5NMBOzu/5VcH85s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=mfJT4MTK; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mfJT4MTK" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-2163bd70069so62937505ad.0 for ; Wed, 18 Dec 2024 00:02:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734508934; x=1735113734; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=APTGqQr2PkEpwaKgWATBkEyab6Gv1pWzvyiVqU1QjxM=; b=mfJT4MTKuMpERe9tR2WJYgxa0dOJa4O3TIjpxXiNHoU47DOWSe/6qSoxQjvKF5FAau FokktAdSAdr/7GR6hyXF67oXa+G4QS2/ImVIABveVQSdLsvOrcvD1i7bj5csSYnJCXJ1 7p0z0ADIa7dWDI+ebm30JBHsrY3FXGfalfLB+dmgoiWdpp6dNqsCJe6qpg0hlLuB/YUZ 9b7Z5LRMH1VrGEdZgSgQg57QMtjCvMzZSpCQGY75H5sY3bv90CAXApV8sPMY7IwzifcY oSfaS3/a1PXwhyJahgC2KTdfVxbKODowXxW9sepUPHzjLam528ocSUVTT9qwIHYVD11f 4w0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734508934; x=1735113734; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=APTGqQr2PkEpwaKgWATBkEyab6Gv1pWzvyiVqU1QjxM=; b=EIKsfq/+SfBqkrSNH+Jit2ng0KBR/oI1G+6zJCul49Ze1gFGyhgi2rhir9rHeyJ6NM hzc285AODWKkRSORaWFS4gGYm/7wtPTGkm4MmKTvDSk1MdWL/uMPTPzl1QPILsXk8TkJ AmCFrw6cg6QKqzmYDIksnM1Tmg5uudehbfXobRPN5VfXGsQovFWs2gOtVzQN0VzS4vFf 73bxIhJ8OzMLFj3kp4bYdgCpEks3WDI48AhMIjeX0+DQZEVm92pqeazyyJSSNe6VF9GM uOky9Ojt3Ir1yBO18//uemZa6UehogjnaJTCM4ERXwdLgdwRtLqyCuR39lfkhBkYiI+6 BbVQ== X-Gm-Message-State: AOJu0YwrnKmlcSBk690QSp5LiOFysaU0TLnFZJjVpeTaQ+EXNT5PNJmM vvc0GgeluOnuJEsMcTHvh0Jeihq28X/awVnn2mEE2f0pYJLXwMok X-Gm-Gg: ASbGncucCByfdzPx3qGtKvz8XH8B0saVPuIz5UwSmNFS1aFCLebRGx7kZ9lkBtb6wJA +TJ3TRm5Yz1kIud66I9J9dMS7ySwemXKaC4IvDKSMvICykhNoXOvIJG5edfFKEoC4aG1VfYlWwY MOHgll+KABOcrDPBSYzCgRjfqNmALYndDllUFrN+mh6wdP/9vR7wL+ph7elUyxhiVjFRutQr2NB Z69197Df9e91VgnoEX8otXNiZR3nqlQjxAidsw56s0vL0odMq9OBOxa0tII6O2V8PDNxOS20aBQ F1hcyGec+g== X-Google-Smtp-Source: AGHT+IECx1ZmRd2aw8C/nMoHl4UXzLUFlbmZHweeGzCyVe1FQZl6iDm722t9AATzPb22rq+GbiS09Q== X-Received: by 2002:a17:903:1105:b0:216:6769:9ee7 with SMTP id d9443c01a7336-218d7247707mr28337635ad.41.1734508933901; Wed, 18 Dec 2024 00:02:13 -0800 (PST) Received: from localhost.localdomain ([103.165.80.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1e5df3csm68718675ad.207.2024.12.18.00.02.09 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 18 Dec 2024 00:02:13 -0800 (PST) From: Hao Jia To: mingo@redhat.com, peterz@infradead.org, mingo@kernel.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com Cc: linux-kernel@vger.kernel.org, Hao Jia Subject: [PATCH] sched/core: Prioritize migrating eligible tasks in sched_balance_rq() Date: Wed, 18 Dec 2024 16:02:03 +0800 Message-Id: <20241218080203.80556-1-jiahao.kernel@gmail.com> X-Mailer: git-send-email 2.39.2 (Apple Git-143) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hao Jia When the PLACE_LAG scheduling feature is enabled and dst_cfs_rq->nr_running is greater than 1, if a task is ineligible (lag < 0) on the source cpu runqueue, it will also be ineligible when it is migrated to the destination cpu runqueue. Because we will keep the original equivalent lag of the task in place_entity(). So if the task was ineligible before, it will still be ineligible after migration. So in sched_balance_rq(), we prioritize migrating eligible tasks, and we soft-limit ineligible tasks, allowing them to migrate only when nr_balance_failed is non-zero to avoid load-balancing trying very hard to balance the load. Below are some benchmark test results. From my test results, this patch shows a slight improvement on hackbench. Benchmark =3D=3D=3D=3D=3D=3D=3D=3D=3D All of the benchmarks are done inside a normal cpu cgroup in a clean environment with cpu turbo disabled, and test machine is: Single NUMA machine model is 13th Gen Intel(R) Core(TM) i7-13700, 12 Core/24 HT. Based on master b86545e02e8c. Results =3D=3D=3D=3D=3D=3D=3D hackbench-process-pipes vanilla patched Amean 1 0.5837 ( 0.00%) 0.5733 ( 1.77%) Amean 4 1.4423 ( 0.00%) 1.4503 ( -0.55%) Amean 7 2.5147 ( 0.00%) 2.4773 ( 1.48%) Amean 12 3.9347 ( 0.00%) 3.8880 ( 1.19%) Amean 21 5.3943 ( 0.00%) 5.3873 ( 0.13%) Amean 30 6.7840 ( 0.00%) 6.6660 ( 1.74%) Amean 48 9.8313 ( 0.00%) 9.6100 ( 2.25%) Amean 79 15.4403 ( 0.00%) 14.9580 ( 3.12%) Amean 96 18.4970 ( 0.00%) 17.9533 ( 2.94%) hackbench-process-sockets vanilla patched Amean 1 0.6297 ( 0.00%) 0.6223 ( 1.16%) Amean 4 2.1517 ( 0.00%) 2.0887 ( 2.93%) Amean 7 3.6377 ( 0.00%) 3.5670 ( 1.94%) Amean 12 6.1277 ( 0.00%) 5.9290 ( 3.24%) Amean 21 10.0380 ( 0.00%) 9.7623 ( 2.75%) Amean 30 14.1517 ( 0.00%) 13.7513 ( 2.83%) Amean 48 24.7253 ( 0.00%) 24.2287 ( 2.01%) Amean 79 43.9523 ( 0.00%) 43.2330 ( 1.64%) Amean 96 54.5310 ( 0.00%) 53.7650 ( 1.40%) tbench4 Throughput vanilla patched Hmean 1 255.97 ( 0.00%) 275.01 ( 7.44%) Hmean 2 511.60 ( 0.00%) 544.27 ( 6.39%) Hmean 4 996.70 ( 0.00%) 1006.57 ( 0.99%) Hmean 8 1646.46 ( 0.00%) 1649.15 ( 0.16%) Hmean 16 2259.42 ( 0.00%) 2274.35 ( 0.66%) Hmean 32 4725.48 ( 0.00%) 4735.57 ( 0.21%) Hmean 64 4411.47 ( 0.00%) 4400.05 ( -0.26%) Hmean 96 4284.31 ( 0.00%) 4267.39 ( -0.39%) Signed-off-by: Hao Jia --- Link to v1: https://lore.kernel.org/all/20241128084858.25220-1-jiahao.kerne= l@gmail.com kernel/sched/fair.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index fbdca89c677f..c4fc2f0cce73 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -9352,6 +9352,30 @@ static inline int migrate_degrades_locality(struct t= ask_struct *p, } #endif =20 +/* + * Check whether the task is ineligible on the destination cpu + * + * When the PLACE_LAG scheduling feature is enabled and + * dst_cfs_rq->nr_running is greater than 1, if the task + * is ineligible, it will also be ineligible when + * it is migrated to the destination cpu. + */ +static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int= dest_cpu) +{ + struct cfs_rq *dst_cfs_rq; + +#if CONFIG_FAIR_GROUP_SCHED + dst_cfs_rq =3D task_group(p)->cfs_rq[dest_cpu]; +#else + dst_cfs_rq =3D &cpu_rq(dest_cpu)->cfs; +#endif + if (sched_feat(PLACE_LAG) && dst_cfs_rq->nr_running && + !entity_eligible(task_cfs_rq(p), &p->se)) + return 1; + + return 0; +} + /* * can_migrate_task - may task p from runqueue rq be migrated to this_cpu? */ @@ -9372,6 +9396,16 @@ int can_migrate_task(struct task_struct *p, struct l= b_env *env) if (throttled_lb_pair(task_group(p), env->src_cpu, env->dst_cpu)) return 0; =20 + /* + * We want to prioritize the migration of eligible tasks. + * For ineligible tasks we soft-limit them and only allow + * them to migrate when nr_balance_failed is non-zero to + * avoid load-balancing trying very hard to balance the load. + */ + if (!env->sd->nr_balance_failed && + task_is_ineligible_on_dst_cpu(p, env->dst_cpu)) + return 0; + /* Disregard percpu kthreads; they are where they need to be. */ if (kthread_is_per_cpu(p)) return 0; --=20 2.34.1