From nobody Mon Feb 9 03:14:05 2026 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 98D223F0761; Wed, 4 Feb 2026 13:27:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770211642; cv=none; b=HUv76Jr+ZQUPEkN/NcBFQVwdSDZQGKo2vTCyNuC71WGqcJw1CJIjMjQo/+DQyiq4iSsFryE3KomiTTLX6T1h6ovXXxb+QcLI5MVErBevGtcV4DifG1ZCXm93PuzElCBL3TLgkNZEwWtFHcpPO081FSN3rEndv54CPB6SjexPgtM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770211642; c=relaxed/simple; bh=rfzGObmIeCxmt81QYQIpwDMneM+OOZMLdqj2+Qy8uek=; h=Date:From:To:Subject:Cc:In-Reply-To:References:MIME-Version: Message-ID:Content-Type; b=pLxf6rD6501jUuXRvARaj+vx1YyrR+vVi3ILKCxrk67tRIxU6Kt/7S60r5DF5CD9TrocnvyCX+ic0nTZNVByGGPdPAfY7jeXGghWjNYgwQia1YUKgYaX8v1AMtGUlQw4AiP4H2D12VPzjSysZuqjByo2ejxlVA2B7VYasxBWSUc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=hIvrR6fu; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=aBI9aQMJ; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="hIvrR6fu"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="aBI9aQMJ" Date: Wed, 04 Feb 2026 13:27:18 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1770211640; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=u+yyqYxvfV2xVI57MuguJ00yKziguG1BXoO8J2ILgnQ=; b=hIvrR6fu0gt2ToesKEX7mXb7S9OfUbS4BIiw5Q9KaRa/oRpWDJ52OfvAGDPfZX4/6g7ccC WwagV4suvtBldeHPaest+y2zTJJYLZw8P/kSpFa0+pG65hFugfn+q2UHPsI3xgBtTkMWmQ GdiBz8GqOghOObEQbPISdqC9Qd7aEsu3QJPZz15+39UvUZOjPepnuokY3nATpuFVJY0tdP OePFTTyAVISH9wUAD753hD/soJjvWvyWH9yH7oAIf9UjVNcoKHlKhMqCIx+UNAKQcM6lbb w+cpNhHCf2/sd+yecN/Cjuo0MFFgCl+VzrhLmHbF5hLgP5eojhw3ZrVM3VzM8g== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1770211640; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=u+yyqYxvfV2xVI57MuguJ00yKziguG1BXoO8J2ILgnQ=; b=aBI9aQMJOud6YO6b3l7oVS/pK8mE65NvSjOjo4PW/xqvrXcNRffGz0YWcpyJihK1DnkgeI vYNlSj+/8sZ0unAg== From: "tip-bot2 for Thomas Gleixner" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: sched/urgent] sched/mmcid: Optimize transitional CIDs when scheduling out Cc: Thomas Gleixner , "Peter Zijlstra (Intel)" , Mathieu Desnoyers , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20260201192835.100194627@kernel.org> References: <20260201192835.100194627@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-ID: <177021163876.2495410.14698078484542619057.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails Precedence: bulk Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable The following commit has been merged into the sched/urgent branch of tip: Commit-ID: 4463c7aa11a6e67169ae48c6804968960c4bffea Gitweb: https://git.kernel.org/tip/4463c7aa11a6e67169ae48c6804968960= c4bffea Author: Thomas Gleixner AuthorDate: Mon, 02 Feb 2026 10:39:55 +01:00 Committer: Peter Zijlstra CommitterDate: Wed, 04 Feb 2026 12:21:12 +01:00 sched/mmcid: Optimize transitional CIDs when scheduling out During the investigation of the various transition mode issues instrumentation revealed that the amount of bitmap operations can be significantly reduced when a task with a transitional CID schedules out after the fixup function completed and disabled the transition mode. At that point the mode is stable and therefore it is not required to drop the transitional CID back into the pool. As the fixup is complete the potential exhaustion of the CID pool is not longer possible, so the CID can be transferred to the scheduling out task or to the CPU depending on the current ownership mode. The racy snapshot of mm_cid::mode which contains both the ownership state and the transition bit is valid because runqueue lock is held and the fixup function of a concurrent mode switch is serialized. Assigning the ownership right there not only spares the bitmap access for dropping the CID it also avoids it when the task is scheduled back in as it directly hits the fast path in both modes when the CID is within the optimal range. If it's outside the range the next schedule in will need to converge so dropping it right away is sensible. In the good case this also allows to go into the fast path on the next schedule in operation. With a thread pool benchmark which is configured to cross the mode switch boundaries frequently this reduces the number of bitmap operations by about 30% and increases the fastpath utilization in the low single digit percentage range. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Mathieu Desnoyers Link: https://patch.msgid.link/20260201192835.100194627@kernel.org --- kernel/sched/sched.h | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index f85fd6b..bd350e4 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -3902,12 +3902,31 @@ static __always_inline void mm_cid_schedin(struct t= ask_struct *next) =20 static __always_inline void mm_cid_schedout(struct task_struct *prev) { + struct mm_struct *mm =3D prev->mm; + unsigned int mode, cid; + /* During mode transitions CIDs are temporary and need to be dropped */ if (likely(!cid_in_transit(prev->mm_cid.cid))) return; =20 - mm_drop_cid(prev->mm, cid_from_transit_cid(prev->mm_cid.cid)); - prev->mm_cid.cid =3D MM_CID_UNSET; + mode =3D READ_ONCE(mm->mm_cid.mode); + cid =3D cid_from_transit_cid(prev->mm_cid.cid); + + /* + * If transition mode is done, transfer ownership when the CID is + * within the convergence range to optimize the next schedule in. + */ + if (!cid_in_transit(mode) && cid < READ_ONCE(mm->mm_cid.max_cids)) { + if (cid_on_cpu(mode)) + cid =3D cid_to_cpu_cid(cid); + + /* Update both so that the next schedule in goes into the fast path */ + mm_cid_update_pcpu_cid(mm, cid); + prev->mm_cid.cid =3D cid; + } else { + mm_drop_cid(mm, cid); + prev->mm_cid.cid =3D MM_CID_UNSET; + } } =20 static inline void mm_cid_switch_to(struct task_struct *prev, struct task_= struct *next)