From nobody Sun Nov 24 23:51:43 2024 Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E7776146596; Fri, 1 Nov 2024 03:18:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730431093; cv=none; b=MbwgtBCHWkzpT37dUX5v942WUng1VyPAuC+wD1A/Fxj2dPWlDL4iMEQ+0heCEVNjntjDS2UNr9DsshALM6PIMhOSbBLGNN1FwS38j6N1xac3VwBtZo1ztOIj384H5LB4gNqeSUbbl/aQ0vsZDVxsmpMfvSwRLkNDb90FRrrHRuk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730431093; c=relaxed/simple; bh=hoBYOBwvRWg95nA71vWiykGK7mKakjibn+uJFK9Yvu8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Fc7EpEHWrnodLGIqi/43HjU3VRcwjrYHvo8WM7tTrhmDl5U+fO9jwBUpSm/t0LHHkLQ8617QL//e9CK9G28Oe2llZa7NsnAEos+v4SIKpzNu1LjsI7KebS7VpBVea6PQEAwFgL+ifeCMnidbFySOxhzD6SeUSQLudi6TK9/CcoU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ID4ph9Pl; arc=none smtp.client-ip=209.85.210.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ID4ph9Pl" Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-720be27db74so930297b3a.1; Thu, 31 Oct 2024 20:18:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730431091; x=1731035891; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ImL5gbpH80k/RQVQHmGfObnVHocW+7l6/wYsDpcrN+I=; b=ID4ph9PlzKe7EONZLkY1A317CJo0634cv63r5UfRS2UIX7jLGYq4qhcvwU1ihmAWqK ZypLn+WzsI7MDaFPbOcJUgreyHOCQunkk9xMuKtJ9xMPl+msqnR1sMutMsljCZTX84QU Hnk6G/0A93E1Ea3a+0wZSDctbPtVeXEcHD6zaX1LoORda1HWKHG7/1UGrteOfzMlnR4c hBLdGGohimlY6L3xmOX4hOG0hRiyyMG706GL+sczPx0I528+A4invdB8ZkF8Ai3fxMOo FS2wK421xMu7bvco4c3UckW+hlmNmisHqxUhWZwjlISMp8yw3ITmQMUDKyiPtjL8kt11 LaMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730431091; x=1731035891; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ImL5gbpH80k/RQVQHmGfObnVHocW+7l6/wYsDpcrN+I=; b=MStRHpO+04qACnoGrukWblFNV+1DQIw2vU7dw5TmdkQSDNlsC+WJoOBcYi7og7lckP LdLrtsrzupl0GQqIFDKCyClbCky16W7QmjK11g0hAS3MddM1OpFwJuQdfNyqIRKDLPif CIJ09Tn2LyCC51hrpZY3IuNQYg9Kr0eKP6B1gONkn9xKcydskTHyu2BkwvQViT++cLvh WX/fk8ExwGfh3QZXYVZhL+kMcbjMvs4+yo1D5mfLc2hTTswXrHDUzwkgwYek5zWQx92/ Bq8TOvNP7U6dISQ9+/Ok1Kc6DwaPI0lvxr1fXDliUkR9VQIfix0vcHHdTZ3YJ5Ti31PL nmsg== X-Forwarded-Encrypted: i=1; AJvYcCVBCZMy/vSCvEx63hvDdOeABYx/MjA/5O6mxNtbqi1Mlu2BaivmToshiRna6lCC1dgW3izEcpm4@vger.kernel.org, AJvYcCVu/Ovbtzai2ryJbSxGu0YyqsbDDdMZ2RASf+I30ehjsnKXwOZrkA82fZy3I7oA8XHcMJ46gkjoZ8+mmExy@vger.kernel.org X-Gm-Message-State: AOJu0YyjPgI0Hp3syGdQchrTFC5SU1d82l1OCQjLeb/DFkz/3n9FUUaq EQr8vp4FNvUsddjIzPe50UL9IZfuvuP3QKrI/sbPTTaaS3Xtn+aLcPgWmle1qvzCNw== X-Google-Smtp-Source: AGHT+IEnuOkWfX4734hR4p6OeZwydqRMrrRS2qoM/A7bvcA7g5CbgJMYrMjsD+XsNHcf/w/9l6Qp5w== X-Received: by 2002:a05:6a20:3544:b0:1d9:761:8ad8 with SMTP id adf61e73a8af0-1db95056f28mr6822894637.21.1730431091084; Thu, 31 Oct 2024 20:18:11 -0700 (PDT) Received: from localhost.localdomain ([39.144.44.206]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-720bc31af9dsm1854445b3a.217.2024.10.31.20.18.06 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 31 Oct 2024 20:18:09 -0700 (PDT) From: Yafang Shao To: mingo@redhat.com, peterz@infradead.org Cc: juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, hannes@cmpxchg.org, surenb@google.com, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yafang Shao Subject: [PATCH v4 1/4] sched: Define sched_clock_irqtime as static key Date: Fri, 1 Nov 2024 11:17:47 +0800 Message-Id: <20241101031750.1471-2-laoar.shao@gmail.com> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20241101031750.1471-1-laoar.shao@gmail.com> References: <20241101031750.1471-1-laoar.shao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Since CPU time accounting is a performance-critical path, let's define sched_clock_irqtime as a static key to minimize potential overhead. Signed-off-by: Yafang Shao --- kernel/sched/cputime.c | 16 +++++++--------- kernel/sched/sched.h | 1 + 2 files changed, 8 insertions(+), 9 deletions(-) diff --git a/kernel/sched/cputime.c b/kernel/sched/cputime.c index 0bed0fa1acd9..d0b6ea737d04 100644 --- a/kernel/sched/cputime.c +++ b/kernel/sched/cputime.c @@ -7,6 +7,8 @@ #include #endif =20 +DEFINE_STATIC_KEY_FALSE(sched_clock_irqtime); + #ifdef CONFIG_IRQ_TIME_ACCOUNTING =20 /* @@ -22,16 +24,14 @@ */ DEFINE_PER_CPU(struct irqtime, cpu_irqtime); =20 -static int sched_clock_irqtime; - void enable_sched_clock_irqtime(void) { - sched_clock_irqtime =3D 1; + static_branch_enable(&sched_clock_irqtime); } =20 void disable_sched_clock_irqtime(void) { - sched_clock_irqtime =3D 0; + static_branch_disable(&sched_clock_irqtime); } =20 static void irqtime_account_delta(struct irqtime *irqtime, u64 delta, @@ -57,7 +57,7 @@ void irqtime_account_irq(struct task_struct *curr, unsign= ed int offset) s64 delta; int cpu; =20 - if (!sched_clock_irqtime) + if (!static_branch_likely(&sched_clock_irqtime)) return; =20 cpu =3D smp_processor_id(); @@ -90,8 +90,6 @@ static u64 irqtime_tick_accounted(u64 maxtime) =20 #else /* CONFIG_IRQ_TIME_ACCOUNTING */ =20 -#define sched_clock_irqtime (0) - static u64 irqtime_tick_accounted(u64 dummy) { return 0; @@ -478,7 +476,7 @@ void account_process_tick(struct task_struct *p, int us= er_tick) if (vtime_accounting_enabled_this_cpu()) return; =20 - if (sched_clock_irqtime) { + if (static_branch_likely(&sched_clock_irqtime)) { irqtime_account_process_tick(p, user_tick, 1); return; } @@ -507,7 +505,7 @@ void account_idle_ticks(unsigned long ticks) { u64 cputime, steal; =20 - if (sched_clock_irqtime) { + if (static_branch_likely(&sched_clock_irqtime)) { irqtime_account_idle_ticks(ticks); return; } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 081519ffab46..038ce65d6635 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -3179,6 +3179,7 @@ struct irqtime { }; =20 DECLARE_PER_CPU(struct irqtime, cpu_irqtime); +DECLARE_STATIC_KEY_FALSE(sched_clock_irqtime); =20 /* * Returns the irqtime minus the softirq time computed by ksoftirqd. --=20 2.43.5 From nobody Sun Nov 24 23:51:43 2024 Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A6F43143890; Fri, 1 Nov 2024 03:18:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730431099; cv=none; b=Hs2LeFh0fy7jxJgfrhiC7CGoSkDrufGNukcgoWMbthZHLE8rjf6Vz35rYwheNNOH3Vhl++6VGQ/Xn3UrOyYwvv/jAWwAf3sVfhEF54IgWQuY73i2kuR5h0ziry8qCFXbMXk7uRvyTFl7KY5TG1PPLtWTX7FitpygqkLT6sjdcdk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730431099; c=relaxed/simple; bh=6kwRE49ETYWaNzkeA8XdNMt4XaICAEAnI7Uy2TCX2Vo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=VgDD8vkwZtYjoZ6SvWGKzeiZw4IWCc+Gs71s/AIEO3gw5VWuysqs1BpLORAxgZ5QX0wzgyL5iQHkzFXYUW5g3BAQppkc+7jyhR44kS4DhV5D1+ziecZWRB68QtXbew2f9dVCFJvtIELqxh7L2shlq/1fdHtcEKVHUXQLF0otGnE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=gDU7bOUT; arc=none smtp.client-ip=209.85.210.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="gDU7bOUT" Received: by mail-pf1-f179.google.com with SMTP id d2e1a72fcca58-72061bfec2dso1302765b3a.2; Thu, 31 Oct 2024 20:18:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730431097; x=1731035897; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PCOOD+oOSZ7Ku2BtoyZC97RAuaceez++PiYgbbH10I8=; b=gDU7bOUTaCFyoeOlc2LiL4fYs6QTZ/DMVvAmXYfgXrZ5DTYQQtlpsYJtE2pwVzr1O+ QALDgzFDorqNvB7+8OM6od7Amgr7wwCbRNzUxzn3tDtL7KWqI0YcMS8rnI4tn3oaZ3QE 8Ohlc4Qv9HCh2XHdgUgGhb9jDjkygq2if/c1BJ5H9oYQzA7UYwZaJSCTzehwa6nAXfGn SbLB4pdw4T5T1NPrlj8vFoFMNLRiH307p7shgDLupud0e7InvOGcjtcwlQRPKVt3th3E IgpAVh3v80TYPAVysy/JDgQo9LB7V1dhCO+IlJAfXP/Wu7Yfpe3qMGn0rcPaAVFLkHgx isGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730431097; x=1731035897; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PCOOD+oOSZ7Ku2BtoyZC97RAuaceez++PiYgbbH10I8=; b=puTcy+SiKRL0rMMH7U30EdDbMRo0xqZlch7AIT/r/D5l6OKXsZTQFwvq0AwZ0GCkaQ GK6Yn+WhxwoTZAy1to4XuVje80y5hsokawb1db+yqzAlmSxS7vm9rtxLU3cNqxtmZ/QS M7+eYMRADyNQzJq8gZMSrL5J47i5Jn/yE60oLjdjmqYKd/M6ePUGB5ea99G0NQnJ4Mzo LFBu66VwN3gwKnZoodvfEjWXAod/lQUN7qIMrlVZCn9lUbvOSCY3FI14+FF1MhxdSW9J UNYoM8G2eEplwMUF8H0iwbV525kQKzpUyqlNhSR07GdGGhFDVYyZH5m0TPHRgvZN3T97 9T7Q== X-Forwarded-Encrypted: i=1; AJvYcCVkmecTrOwLOfcHLtWDUi7ialfxGNdrqRM2kRCQO237EYdQOSfNaEnzOfr05ERf0nLgpOqwtjPu1V71zxkf@vger.kernel.org, AJvYcCVv+WAsL7MHbZUwq1KeOoAJT4UbxoozmRpcfvnc2r+FPD85Ri1dQygZ8YxE4p9KBdCtwL9I2rac@vger.kernel.org X-Gm-Message-State: AOJu0Yxl0UwJQt5gxWT7+mzaWOSmW1ByBG2kAXdSmZE5Puv0LFs0Tsnb uvzPNz79dqX+BzlX6M/IJNMr6xFv/qQXAV+xC4BHdmiQfafGw8T0 X-Google-Smtp-Source: AGHT+IFp3oIRnkmXbG2waFsi4w5plsLbXBIX8s+1Wwg/DGPH2uReRuRsOAEKPmyk8MHm4Z7xw3lt7Q== X-Received: by 2002:a05:6a21:e88:b0:1d2:ef5c:13f6 with SMTP id adf61e73a8af0-1dba54e0957mr2097436637.34.1730431096781; Thu, 31 Oct 2024 20:18:16 -0700 (PDT) Received: from localhost.localdomain ([39.144.44.206]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-720bc31af9dsm1854445b3a.217.2024.10.31.20.18.11 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 31 Oct 2024 20:18:15 -0700 (PDT) From: Yafang Shao To: mingo@redhat.com, peterz@infradead.org Cc: juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, hannes@cmpxchg.org, surenb@google.com, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yafang Shao Subject: [PATCH v4 2/4] sched: Don't account irq time if sched_clock_irqtime is disabled Date: Fri, 1 Nov 2024 11:17:48 +0800 Message-Id: <20241101031750.1471-3-laoar.shao@gmail.com> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20241101031750.1471-1-laoar.shao@gmail.com> References: <20241101031750.1471-1-laoar.shao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" sched_clock_irqtime may be disabled due to the clock source, in which case IRQ time should not be accounted. Let's add a conditional check to avoid unnecessary logic. Signed-off-by: Yafang Shao --- kernel/sched/core.c | 44 +++++++++++++++++++++++--------------------- 1 file changed, 23 insertions(+), 21 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index dbfb5717d6af..06a06f0897c3 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -740,29 +740,31 @@ static void update_rq_clock_task(struct rq *rq, s64 d= elta) s64 __maybe_unused steal =3D 0, irq_delta =3D 0; =20 #ifdef CONFIG_IRQ_TIME_ACCOUNTING - irq_delta =3D irq_time_read(cpu_of(rq)) - rq->prev_irq_time; + if (static_branch_likely(&sched_clock_irqtime)) { + irq_delta =3D irq_time_read(cpu_of(rq)) - rq->prev_irq_time; =20 - /* - * Since irq_time is only updated on {soft,}irq_exit, we might run into - * this case when a previous update_rq_clock() happened inside a - * {soft,}IRQ region. - * - * When this happens, we stop ->clock_task and only update the - * prev_irq_time stamp to account for the part that fit, so that a next - * update will consume the rest. This ensures ->clock_task is - * monotonic. - * - * It does however cause some slight miss-attribution of {soft,}IRQ - * time, a more accurate solution would be to update the irq_time using - * the current rq->clock timestamp, except that would require using - * atomic ops. - */ - if (irq_delta > delta) - irq_delta =3D delta; + /* + * Since irq_time is only updated on {soft,}irq_exit, we might run into + * this case when a previous update_rq_clock() happened inside a + * {soft,}IRQ region. + * + * When this happens, we stop ->clock_task and only update the + * prev_irq_time stamp to account for the part that fit, so that a next + * update will consume the rest. This ensures ->clock_task is + * monotonic. + * + * It does however cause some slight miss-attribution of {soft,}IRQ + * time, a more accurate solution would be to update the irq_time using + * the current rq->clock timestamp, except that would require using + * atomic ops. + */ + if (irq_delta > delta) + irq_delta =3D delta; =20 - rq->prev_irq_time +=3D irq_delta; - delta -=3D irq_delta; - delayacct_irq(rq->curr, irq_delta); + rq->prev_irq_time +=3D irq_delta; + delta -=3D irq_delta; + delayacct_irq(rq->curr, irq_delta); + } #endif #ifdef CONFIG_PARAVIRT_TIME_ACCOUNTING if (static_key_false((¶virt_steal_rq_enabled))) { --=20 2.43.5 From nobody Sun Nov 24 23:51:43 2024 Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2A575143C72; Fri, 1 Nov 2024 03:18:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730431105; cv=none; b=ZwDpBn9viWMIPcX1JI0NoqZRSPlNfOLEcIrFfB4JRwr/R55EVNo2SnruabonzRqDKU4SbCJOhMeDjBK1D1ZLQISMkx99a5UvFL9Sj0hFGxNv7hG1iwm5T+26l0JhkxxD+0u2s8TVnRWToYzymQovORbR5Kq/ewGl0oMaLrjh5IY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730431105; c=relaxed/simple; bh=r8YcZ0dpdXNp/7S1zdAyAeQPX/2DPYDPXuaHe/RSVM4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=fvqWX9BbPkEK8XQHa+j9q+f3KUCRPiJoXrm4c/6q/L7+gTNsU1MMCCsRzQxEw7ovGQXS4XrCK7wxQ2buyOQEoylPEp5VNKWVd28w2JNNg/ymA0XECbheFXGsqu4+HrY6kIP8V0ZlMRCSyNbSkzNekKBWYi7MDN5gTyz4guF+Wl8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=g7bLsg96; arc=none smtp.client-ip=209.85.210.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="g7bLsg96" Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-720b2d8bb8dso1225707b3a.1; Thu, 31 Oct 2024 20:18:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730431102; x=1731035902; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UUgCvhOKE3smcEC280xZ+zWa212yApUb0NoA+1fbR/I=; b=g7bLsg96DuQ+uGEPUwHuy2OpW5Xtlx3mkHll430Xwut/BFaFkYbodpvQygoq2om3GM NYm5oxiq11iVD9IBX3zZcInWxdCPQ2QvXEkI2W1iM1LnSBpp8gBu6yvtBrpzUnyq3zzm zAqlNiZDQ6O/92Ad+VOqxIdHAUP2KBdzGidN+vYszt5UTS2Wnnk5z4EWrOL7yVpAJou3 FA6hzwAzy923fA8hh0i42qe2sL8EnOOCHtVWWeR+XalJV2TZHeZqQHwdd7U2zec1FLA5 hwwCtnZrIzZrdLNW1rkrKxsaLDxztSGQV5YJYmLOYGgOt3NAYhZVqFBi7z+4BPNGOXIH qOhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730431102; x=1731035902; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UUgCvhOKE3smcEC280xZ+zWa212yApUb0NoA+1fbR/I=; b=SuuOSZhktBZayVLjQRQH2jz2BSjoKck2rM4Wb9FhOVJjnyC5FsLwkxplur52oSAmns F17nkXAefYuZLfvH88DS20U5vcPkQR094bfsDb+iSaDxTXnBEkKGtMVx5w86a2QPOZ6+ qUtA0ipr6pT68kvAI+tnfzlYrgaE0vq6SynBIp5AKPdoWhl/zDFRFDXASSjOs6agftjB f3YQr83K3aOlXmrkik2z+qDwFblfPAAcuh6O7K+7uVAnkuRa8EhjP65p+ZLFIstUVqQu oopQnw9ghZmdHa9eGGBL0zIuY5TrlR8N8wvPMIcbi0y4IwzmXIogZPtDq5Yt7qvY99js yBIA== X-Forwarded-Encrypted: i=1; AJvYcCUa4p+zl7hW9REIIBdqW5gmEMJ5gN2X0UfqMUlGumn2xX8senlXxyHFBZgZSjk1vvKChKOFMqbHv9G9B/EE@vger.kernel.org, AJvYcCWwy3wtgKAaYpsk3H3ZT3qd7k4zPUfyKCxYlmtXKPH9eNmuOquBS+Egb0KvacnD3VgbDliKA+T7@vger.kernel.org X-Gm-Message-State: AOJu0Yyc0MbD87NCFYKbnB0vXjSvTKbUnrUI4iK2dZPHRwsIUYa+OcTi tmRQ0w5wegjggQHiwfMPcH0AVfIHgko2F/nGJcmvGKfnLr4/95d4 X-Google-Smtp-Source: AGHT+IFerp3i9PMi0KaUp00Djg4A3gCBpMNYApZcLGXOBnYchDqJbxy1LByYRd7ppnrDP3jVK8UcrA== X-Received: by 2002:a05:6a20:b40b:b0:1d9:651:7d6c with SMTP id adf61e73a8af0-1d9a84d97famr29656569637.38.1730431102334; Thu, 31 Oct 2024 20:18:22 -0700 (PDT) Received: from localhost.localdomain ([39.144.44.206]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-720bc31af9dsm1854445b3a.217.2024.10.31.20.18.17 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 31 Oct 2024 20:18:21 -0700 (PDT) From: Yafang Shao To: mingo@redhat.com, peterz@infradead.org Cc: juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, hannes@cmpxchg.org, surenb@google.com, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yafang Shao Subject: [PATCH v4 3/4] sched, psi: Don't account irq time if sched_clock_irqtime is disabled Date: Fri, 1 Nov 2024 11:17:49 +0800 Message-Id: <20241101031750.1471-4-laoar.shao@gmail.com> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20241101031750.1471-1-laoar.shao@gmail.com> References: <20241101031750.1471-1-laoar.shao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" sched_clock_irqtime may be disabled due to the clock source. When disabled, irq_time_read() won't change over time, so there is nothing to account. We can save iterating the whole hierarchy on every tick and context switch. Signed-off-by: Yafang Shao Acked-by: Johannes Weiner --- kernel/sched/psi.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/kernel/sched/psi.c b/kernel/sched/psi.c index 84dad1511d1e..4d26a106f03b 100644 --- a/kernel/sched/psi.c +++ b/kernel/sched/psi.c @@ -998,7 +998,8 @@ void psi_account_irqtime(struct rq *rq, struct task_str= uct *curr, struct task_st s64 delta; u64 irq; =20 - if (static_branch_likely(&psi_disabled)) + if (static_branch_likely(&psi_disabled) || + !static_branch_likely(&sched_clock_irqtime)) return; =20 if (!curr->pid) --=20 2.43.5 From nobody Sun Nov 24 23:51:43 2024 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1504C146000; Fri, 1 Nov 2024 03:18:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730431111; cv=none; b=YhmBbGfX/HEqiamoQCR1Dj28NiwMtWPLjDdzPrszJVq9rH2tpWztXrJUeOV1THq4AnsFSuX7bKCVXrQdmIP5HcGtWue2zePhOyGcgXm//FJ/982g138NhZqwlIm3FxGjjd/62j7RaGYQxm+A6QWCRdC9E1Aw1L1m35fmGJwHhkA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730431111; c=relaxed/simple; bh=C4kHVk8yGOp97R24xFKPOimMNvvcGIgopcVjcE0Zn/0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=nnyAsCCWWURnAOYvU08x8VbAAIZnsYfW9AU3CiLpav2Fwp3kQ5xYlVjCkOvftqDE3Ffszkj3RFRBo/u7OPEiC5Jg2ohRgJi5kEQ+luYsqf3uxXp3glrYZk/8zJParEI0d1/LsdIw1wDNjh5IK9uICEDo3wo+e6wSPH5T3IwtP6w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=dC5kQXZE; arc=none smtp.client-ip=209.85.210.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dC5kQXZE" Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-720d01caa66so204954b3a.2; Thu, 31 Oct 2024 20:18:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730431107; x=1731035907; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=D2vaxuzgksXz4yWA/68x9ijGVuyFQOHqmQ3f7bHhuGo=; b=dC5kQXZE4KOjm9zD//QXu1If6OrS33nItIMTT7P4T5I0+niQl4ZeMRCR02rSkIfRPn QcKn1+Y6zUmYUpbtsBHJYT68O+wVHkPgZt81ThQ19Ekt3G8VkFwAcqNzxkrAxlo5/Ed0 97HNVKNAgZyfn+lLKVpVbcfupZTIthiWBweEwqB2ioRX4jEYu/cDf3HaJJNYVJZZdPse XRpeTWaEshxQdaAl7vTE6wducpqtOQfN/mF/JShOTJ1aJD9Ztk/U0m1yLGgXCg8xwXlI UTk5K+r6yavM8yuxc4Z7ReUPp9Uox5ufISCC4bgz3dIBC5tif2yYW8aNA2BoWnqVQ1RY Db9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730431107; x=1731035907; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=D2vaxuzgksXz4yWA/68x9ijGVuyFQOHqmQ3f7bHhuGo=; b=VibMGK7atRBO2hMYnP8O5sqm1xrTeH5uBKrS6AJoVj30FctmoBLJkMxTNtvYz1mI/4 7pASKp2kSbEWkbVXQAHx6g9T1GzTQtnLh6YPIiZqYEBz7bvpluZcOODSxUhQqBofZQlz Qo1QnGJ79q2WaJMWVThKINCjCJf8HcCBclrpnd97UTrPQMVKUBUSFBj+JoQ6/xlAXGIq vvHVzi2pMP2yNzdlbp6PscMW5EX22GBZOmgGVChLlX6VSqUuJKZX5NvdZBa/X4aElSkz t5+n5Kmej4kQzCH9GmmxA3lp3ORKWcqCDGn2xoH5vKKzHLp0kgsJjwHUsGlSsHs7XnK/ 76jw== X-Forwarded-Encrypted: i=1; AJvYcCVLzQcFfCANBK7Rb4pm6hyMjqvBNRrteUI5BWajlGm7C04MAjGdHrpCA/l77VuSat3Y+5mciY1b@vger.kernel.org, AJvYcCX/pq720R3+bT5Lqpaz1hy7Pa/ZyWFtPvqdKi/doF5uG3QeO0sIo9VIw+Dg0qbzGYQL8tHilJujUUogifUe@vger.kernel.org X-Gm-Message-State: AOJu0YzQYHDPnI8hbuILZ4VkagVgb8h9OfRsI7lT2IFPwDulLjIJ+z+o FiEblUxzHFIATWqn8nDRbHh0MwABnN8hmrbMuKr6F+1EzSvWLWaKEXPCYe+dyrBRtw== X-Google-Smtp-Source: AGHT+IGkEDI6zIr7JO0DhH4ibPacIo0wpRogVp3awtFen42qJnMvrwJkCvWL8v+TAtCOElaKOipqMQ== X-Received: by 2002:a05:6a00:2e22:b0:71e:6fcb:7688 with SMTP id d2e1a72fcca58-720b9ddb773mr7326970b3a.25.1730431107149; Thu, 31 Oct 2024 20:18:27 -0700 (PDT) Received: from localhost.localdomain ([39.144.44.206]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-720bc31af9dsm1854445b3a.217.2024.10.31.20.18.22 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 31 Oct 2024 20:18:26 -0700 (PDT) From: Yafang Shao To: mingo@redhat.com, peterz@infradead.org Cc: juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, hannes@cmpxchg.org, surenb@google.com, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yafang Shao Subject: [PATCH v4 4/4] sched: Fix cgroup irq accounting for CONFIG_IRQ_TIME_ACCOUNTING Date: Fri, 1 Nov 2024 11:17:50 +0800 Message-Id: <20241101031750.1471-5-laoar.shao@gmail.com> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20241101031750.1471-1-laoar.shao@gmail.com> References: <20241101031750.1471-1-laoar.shao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable After enabling CONFIG_IRQ_TIME_ACCOUNTING to monitor IRQ pressure in our container environment, we observed several noticeable behavioral changes. One of our IRQ-heavy services, such as Redis, reported a significant reduction in CPU usage after upgrading to the new kernel with CONFIG_IRQ_TIME_ACCOUNTING enabled. However, despite adding more threads to handle an increased workload, the CPU usage could not be raised. In other words, even though the container=E2=80=99s CPU usage appeared low, it= was unable to process more workloads to utilize additional CPU resources, which caused issues. This behavior can be demonstrated using netperf: function start_server() { for j in `seq 1 3`; do netserver -p $[12345+j] > /dev/null & done } server_ip=3D$1 function start_client() { # That applies to cgroup2 as well. mkdir -p /sys/fs/cgroup/cpuacct/test echo $$ > /sys/fs/cgroup/cpuacct/test/cgroup.procs for j in `seq 1 3`; do port=3D$[12345+j] taskset -c 0 netperf -H ${server_ip} -l ${run_time:-30000} \ -t TCP_STREAM -p $port -- -D -m 1k -M 1K -s 8k -S 8k \ > /dev/null & done } start_server start_client We can verify the CPU usage of the test cgroup using cpuacct.stat. The output shows: system: 53 user: 2 The CPU usage of the cgroup is relatively low at around 55%, but this usage doesn't increase, even with more netperf tasks. The reason is that CPU0 is at 100% utilization, as confirmed by mpstat: 02:56:22 PM CPU %usr %nice %sys %iowait %irq %soft %steal = %guest %gnice %idle 02:56:23 PM 0 0.99 0.00 55.45 0.00 0.99 42.57 0.00 = 0.00 0.00 0.00 02:56:23 PM CPU %usr %nice %sys %iowait %irq %soft %steal = %guest %gnice %idle 02:56:24 PM 0 2.00 0.00 55.00 0.00 0.00 43.00 0.00 = 0.00 0.00 0.00 It is clear that the %soft is not accounted into the cgroup of the interrupted task. This behavior is unexpected. We should account for IRQ time to the cgroup to reflect the pressure the group is under. After a thorough analysis, I discovered that this change in behavior is due to commit 305e6835e055 ("sched: Do not account irq time to current task"), which altered whether IRQ time should be charged to the interrupted task. While I agree that a task should not be penalized by random interrupts, the task itself cannot progress while interrupted. Therefore, the interrupted time should be reported to the user. The system metric in cpuacct.stat is crucial in indicating whether a container is under heavy system pressure, including IRQ/softirq activity. Hence, IRQ/softirq time should be accounted for in the cpuacct system usage, which also applies to cgroup2=E2=80=99s rstat. This patch reintroduces IRQ/softirq accounting to cgroups. Signed-off-by: Yafang Shao Cc: Johannes Weiner --- kernel/sched/core.c | 33 +++++++++++++++++++++++++++++++-- kernel/sched/psi.c | 14 +++----------- kernel/sched/stats.h | 7 ++++--- 3 files changed, 38 insertions(+), 16 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 06a06f0897c3..5ed2c5c8c911 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -5579,6 +5579,35 @@ __setup("resched_latency_warn_ms=3D", setup_resched_= latency_warn_ms); static inline u64 cpu_resched_latency(struct rq *rq) { return 0; } #endif /* CONFIG_SCHED_DEBUG */ =20 +#ifdef CONFIG_IRQ_TIME_ACCOUNTING +static void account_irqtime(struct rq *rq, struct task_struct *curr, + struct task_struct *prev) +{ + int cpu =3D smp_processor_id(); + s64 delta; + u64 irq; + + if (!static_branch_likely(&sched_clock_irqtime)) + return; + + irq =3D irq_time_read(cpu); + delta =3D (s64)(irq - rq->psi_irq_time); + if (delta < 0) + return; + + rq->psi_irq_time =3D irq; + psi_account_irqtime(rq, curr, prev, delta); + cgroup_account_cputime(curr, delta); + /* We account both softirq and irq into softirq */ + cgroup_account_cputime_field(curr, CPUTIME_SOFTIRQ, delta); +} +#else +static inline void account_irqtime(struct rq *rq, struct task_struct *curr, + struct task_struct *prev) +{ +} +#endif + /* * This function gets called by the timer code, with HZ frequency. * We call it with interrupts disabled. @@ -5600,7 +5629,7 @@ void sched_tick(void) rq_lock(rq, &rf); =20 curr =3D rq->curr; - psi_account_irqtime(rq, curr, NULL); + account_irqtime(rq, curr, NULL); =20 update_rq_clock(rq); hw_pressure =3D arch_scale_hw_pressure(cpu_of(rq)); @@ -6683,7 +6712,7 @@ static void __sched notrace __schedule(int sched_mode) ++*switch_count; =20 migrate_disable_switch(rq, prev); - psi_account_irqtime(rq, prev, next); + account_irqtime(rq, prev, next); psi_sched_switch(prev, next, block); =20 trace_sched_switch(preempt, prev, next, prev_state); diff --git a/kernel/sched/psi.c b/kernel/sched/psi.c index 4d26a106f03b..1adb41b2ae1d 100644 --- a/kernel/sched/psi.c +++ b/kernel/sched/psi.c @@ -990,16 +990,14 @@ void psi_task_switch(struct task_struct *prev, struct= task_struct *next, } =20 #ifdef CONFIG_IRQ_TIME_ACCOUNTING -void psi_account_irqtime(struct rq *rq, struct task_struct *curr, struct t= ask_struct *prev) +void psi_account_irqtime(struct rq *rq, struct task_struct *curr, struct t= ask_struct *prev, + s64 delta) { int cpu =3D task_cpu(curr); struct psi_group *group; struct psi_group_cpu *groupc; - s64 delta; - u64 irq; =20 - if (static_branch_likely(&psi_disabled) || - !static_branch_likely(&sched_clock_irqtime)) + if (static_branch_likely(&psi_disabled)) return; =20 if (!curr->pid) @@ -1010,12 +1008,6 @@ void psi_account_irqtime(struct rq *rq, struct task_= struct *curr, struct task_st if (prev && task_psi_group(prev) =3D=3D group) return; =20 - irq =3D irq_time_read(cpu); - delta =3D (s64)(irq - rq->psi_irq_time); - if (delta < 0) - return; - rq->psi_irq_time =3D irq; - do { u64 now; =20 diff --git a/kernel/sched/stats.h b/kernel/sched/stats.h index 767e098a3bd1..17eefe5876a5 100644 --- a/kernel/sched/stats.h +++ b/kernel/sched/stats.h @@ -111,10 +111,11 @@ void psi_task_change(struct task_struct *task, int cl= ear, int set); void psi_task_switch(struct task_struct *prev, struct task_struct *next, bool sleep); #ifdef CONFIG_IRQ_TIME_ACCOUNTING -void psi_account_irqtime(struct rq *rq, struct task_struct *curr, struct t= ask_struct *prev); +void psi_account_irqtime(struct rq *rq, struct task_struct *curr, + struct task_struct *prev, s64 delta); #else static inline void psi_account_irqtime(struct rq *rq, struct task_struct *= curr, - struct task_struct *prev) {} + struct task_struct *prev, s64 delta) {} #endif /*CONFIG_IRQ_TIME_ACCOUNTING */ /* * PSI tracks state that persists across sleeps, such as iowaits and @@ -215,7 +216,7 @@ static inline void psi_sched_switch(struct task_struct = *prev, struct task_struct *next, bool sleep) {} static inline void psi_account_irqtime(struct rq *rq, struct task_struct *= curr, - struct task_struct *prev) {} + struct task_struct *prev, s64 delta) {} #endif /* CONFIG_PSI */ =20 #ifdef CONFIG_SCHED_INFO --=20 2.43.5