From nobody Sat Feb 7 18:15:41 2026 Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D4B2D4964D for ; Mon, 19 Aug 2024 15:43:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724082189; cv=none; b=bvMFw7nlkQ91RPYaR/gVB/3A5Au1+955D58/qUuIuvpAZM1sAtKO/hO3PGCLQ6Q+jx3U5Oo1prcuC5rmSfxROGo9ppn/1z1Li0zMiO12dAXYaNdHg5jZ9qUO4CxzH/k+iNybkGmM4FlVpT9I4JF276rA89BU/AZX93DcGUpR4S4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724082189; c=relaxed/simple; bh=K0xjumLwWN6l2imOx3JvlkgsS/HBAmMUKFbMV39pZxQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hLwBDS7qFT+wwJpFzts/yId+Kru6uiyoDKOGjUJVCMizLgRgVkM7P6VD0eX2Ktwx+w8Ijn8EJyMcaCaGVcLLW1MtKJPDzhgGf/DWYdXH6oum8nfIAON97SJO9Vy9gzvcsEs7+3iIrdr2XYs5r13YtEWLlI/K4ZiwreydXZLO13o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=3L3KlOZO; arc=none smtp.client-ip=209.85.216.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="3L3KlOZO" Received: by mail-pj1-f50.google.com with SMTP id 98e67ed59e1d1-2d3f39e7155so561675a91.0 for ; Mon, 19 Aug 2024 08:43:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1724082185; x=1724686985; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JkqFfu8TNNTkeL8UJk+hasytcQg+Y2ER8tyNsGPxyto=; b=3L3KlOZOljEBFK7cqNPgZX8S8TsfH13v0zRaivILOV98UEqq4NvPac3J57FyYUhrzi JzVtb5oZksFiAS9cPqbn78SPX+qDXHXLENECHbYKRp9w24nPbByT/hfgAIwNclJAVAzp Mjo95BCbKrk0Liz2EQH0+0ezQVmR8ucPtOaLjNg4dU3LJARayeV35braKuNceHJYGL9b ypEKHdPD9mX8Wk5t0g8zpBgnLVfHMhFbWBNZJ30iRni5LrNRF9o2KJ7I3Cin/sxmys8x JEHBYQP2FoVXYl5T6scRVXUnLDoYLXNmAp4a2rt1PTBkxVLNdE+D6ue6kuZKZ8EFMfxL O/Eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724082185; x=1724686985; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JkqFfu8TNNTkeL8UJk+hasytcQg+Y2ER8tyNsGPxyto=; b=g9NzgRAUCRaSMSTmwouSj1tHB+XT8QFqxYb3jWYCDV8HBMYf5WNZpuAXZNyBsCaS4K kcs58G6DJzSjXbsFVPnjT5UuOI+aPRZGRsjzkDgdP7xWPQKSimgnve9spYtJDKclNnrx AszXmnehL2WQXNEcFKAWffaucBtDlY7FpAvQ8FgDZOm33vBS5J3ouoKMy4BX6+QD01Sq TwiiPLfdpE7sjjF/wPD+xE+B3eQhIMihPJwRFjkNABvmPQv6xbGnYC44Ow6t4Roea1Cw NPg/nLjzaIIzpdLLLPGzPZiUrULYAKpZlaRQQ+UxEmxwQuura7cngtpA8/ZHXPyyEcYl uvZA== X-Gm-Message-State: AOJu0YwN1g0DBzmG9deCtGG95thkqkPHjpcX8Ok8hmNp4dXVgVyc/nOG NinGLpMSkXFPJ2BZPP7OEdPLdFSAdxoHaHBfTB3yJHYK1CvYE0E5jfRrVfc9cTwsRg8qmOPc1z0 s X-Google-Smtp-Source: AGHT+IHehYd4v6KtW8X7VGSGbhwu5p208ySNI/IH/HH5ydnbOd4lF7D/RhXDCtJt2VCxomuthnjLYw== X-Received: by 2002:a17:90b:146:b0:2c8:4623:66cd with SMTP id 98e67ed59e1d1-2d3e1a42e70mr7456254a91.1.1724082184746; Mon, 19 Aug 2024 08:43:04 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d3e3c751f1sm7356041a91.38.2024.08.19.08.43.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Aug 2024 08:43:03 -0700 (PDT) From: Jens Axboe To: linux-kernel@vger.kernel.org Cc: peterz@infradead.org, tglx@linutronix.de, Jens Axboe Subject: [PATCH 1/4] sched/core: add helpers for iowait handling Date: Mon, 19 Aug 2024 09:39:46 -0600 Message-ID: <20240819154259.215504-2-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240819154259.215504-1-axboe@kernel.dk> References: <20240819154259.215504-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Adds helpers to inc/dec the runqueue iowait count, based on the task, and use those in the spots where the count is manipulated. Adds an rq_iowait() helper, to abstract out how the per-rq stats are read. No functional changes in this patch, just in preparation for switching the type of 'nr_iowait'. Signed-off-by: Jens Axboe --- kernel/sched/core.c | 23 +++++++++++++++++++---- kernel/sched/cputime.c | 3 +-- kernel/sched/sched.h | 2 ++ 3 files changed, 22 insertions(+), 6 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index ab50100363ca..9bf1b67818d0 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3541,6 +3541,21 @@ static inline bool rq_has_pinned_tasks(struct rq *rq) =20 #endif /* !CONFIG_SMP */ =20 +static void task_iowait_inc(struct task_struct *p) +{ + atomic_inc(&task_rq(p)->nr_iowait); +} + +static void task_iowait_dec(struct task_struct *p) +{ + atomic_dec(&task_rq(p)->nr_iowait); +} + +int rq_iowait(struct rq *rq) +{ + return atomic_read(&rq->nr_iowait); +} + static void ttwu_stat(struct task_struct *p, int cpu, int wake_flags) { @@ -3607,7 +3622,7 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p= , int wake_flags, #endif if (p->in_iowait) { delayacct_blkio_end(p); - atomic_dec(&task_rq(p)->nr_iowait); + task_iowait_dec(p); } =20 activate_task(rq, p, en_flags); @@ -4184,7 +4199,7 @@ int try_to_wake_up(struct task_struct *p, unsigned in= t state, int wake_flags) if (task_cpu(p) !=3D cpu) { if (p->in_iowait) { delayacct_blkio_end(p); - atomic_dec(&task_rq(p)->nr_iowait); + task_iowait_dec(p); } =20 wake_flags |=3D WF_MIGRATED; @@ -5282,7 +5297,7 @@ unsigned long long nr_context_switches(void) =20 unsigned int nr_iowait_cpu(int cpu) { - return atomic_read(&cpu_rq(cpu)->nr_iowait); + return rq_iowait(cpu_rq(cpu)); } =20 /* @@ -6512,7 +6527,7 @@ static void __sched notrace __schedule(unsigned int s= ched_mode) deactivate_task(rq, prev, DEQUEUE_SLEEP | DEQUEUE_NOCLOCK); =20 if (prev->in_iowait) { - atomic_inc(&rq->nr_iowait); + task_iowait_inc(prev); delayacct_blkio_start(); } } diff --git a/kernel/sched/cputime.c b/kernel/sched/cputime.c index 0bed0fa1acd9..b826267714de 100644 --- a/kernel/sched/cputime.c +++ b/kernel/sched/cputime.c @@ -222,9 +222,8 @@ void account_steal_time(u64 cputime) void account_idle_time(u64 cputime) { u64 *cpustat =3D kcpustat_this_cpu->cpustat; - struct rq *rq =3D this_rq(); =20 - if (atomic_read(&rq->nr_iowait) > 0) + if (rq_iowait(this_rq()) > 0) cpustat[CPUTIME_IOWAIT] +=3D cputime; else cpustat[CPUTIME_IDLE] +=3D cputime; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 1e1d1b467af2..b6b3b565bcb1 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -3579,6 +3579,8 @@ static inline void init_sched_mm_cid(struct task_stru= ct *t) { } extern u64 avg_vruntime(struct cfs_rq *cfs_rq); extern int entity_eligible(struct cfs_rq *cfs_rq, struct sched_entity *se); =20 +int rq_iowait(struct rq *rq); + #ifdef CONFIG_RT_MUTEXES =20 static inline int __rt_effective_prio(struct task_struct *pi_task, int pri= o) --=20 2.43.0 From nobody Sat Feb 7 18:15:41 2026 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 251BA82D94 for ; Mon, 19 Aug 2024 15:43:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724082190; cv=none; b=d49W/JdtyaxVMWOPKgp+xietF7wsmfEIKo8eFMMznAfGVIy5jPFHjm5trXY7tAlJjNHcl004/ZEl5iMdq4XfbA9YsO4BR+h6UJKOPsryjS4Pir1TTFqgcedM6NNuUQE7U9+fRiEMAsY29awnzyMR3KRnrDdTS0jtddG9HV5WEGo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724082190; c=relaxed/simple; bh=UTfeDrPv1P7VGVLIn2EsldU3QCEyihFMAXquA3OWvIc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ACxwVLB3rkznsK21VTNC+vZ9gDum3Hg6MKuzfm0r1eZs3AJJzGJam6l1NS6jWQl0ARYc4rQw1OhG2sQ6Lz/6U5e7nwqEsinlVcyxBJnIYclLhB5wNRkJIer2G5ifWRZKrkNsLwjeXiXZWOsW0VI85dX7urBZBmDnY5GBvjohzRk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=N5fO/SQU; arc=none smtp.client-ip=209.85.216.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="N5fO/SQU" Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-2d3ba4e6ad9so849256a91.2 for ; Mon, 19 Aug 2024 08:43:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1724082186; x=1724686986; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4QZ5nGY8yRhaSovqTVP5eDgWCjwMJ5zw2jKCQIUNZGI=; b=N5fO/SQUnWffdEuGy0NOnkOfEePCeB+eJF9GEg0tYBaNkto2PnZUu7Us0+YZdYE7mP 5GHeU1g9wLAlVL1+VhFES4dc6Lcl0Yvuag8462xBchOrbgJcoKpn6BelYcrCCJe5MEdl pa5R8ONemniaQzdz/2vjo+p6bKu/SxUZqiew55OH2g+GcTeu/TdzJU02Yr0ZQ9DWJYcV Hg/dqgW05OKMCRTXDjhyctp+zDNqQmgzWSpusI/JQ0bm7878zHJ6YzkPwJ5X9nbDlMKg Xgqlgm8Ow+zPjImW9mSK5/uDZizcm91zwZPojrTukvgdTzDTKTEHL5lg6JbeAYyubU6t Uu2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724082186; x=1724686986; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4QZ5nGY8yRhaSovqTVP5eDgWCjwMJ5zw2jKCQIUNZGI=; b=Qy454xVImn8GxLL3oyUi9k60Aee1NhlL/WgLq8AX6bBmYDiaGyBAPjS2hJdOnO7pkU +gqOojgpHeK4hrqr0SPrkUYnWWd8efSAjwYfDsD2k4+YKJ5GoT+bZycU3w7nIDHZhaG8 4feWZ4RDRsBv9JsUSIspFO7nks11JeKuhxzkA13mS+wS1eBZpsA0WY+W13Qvf5KJktdI lCibJHMfmce86+QqPPXV5o2TsbeyfAfILv59qtf87RlhnJKpMg1Lnu2CGIAHe1A8lQDv a7gUJKi+jFKZU9SdBQVYnlXI6MwUNONEYJXMYbzYk6GXJqsbMbtHOOQLKZchjhzf4tQA JhQw== X-Gm-Message-State: AOJu0YyGCKc7s2UtGjb8O5rUaiy87UK0HBgOuQ0Ki1zjM+dgGjarQJfh EkJLfHjZHs89eOwq2bq1Oq6t4nn6y1NvPabG88IcYnbg2pAsl/h+1anyvtFysne+NjS0Xirte5s 2 X-Google-Smtp-Source: AGHT+IGgQiHQ09ncrftjT78NhUCy+9R5076zqxCfZlDz/0Wkj1fDTUeRrbdX3LqF2lpcK0t+nWMiBQ== X-Received: by 2002:a17:90a:fa8c:b0:2d3:cab7:f1dd with SMTP id 98e67ed59e1d1-2d3dfc396d7mr7214984a91.1.1724082186329; Mon, 19 Aug 2024 08:43:06 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d3e3c751f1sm7356041a91.38.2024.08.19.08.43.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Aug 2024 08:43:05 -0700 (PDT) From: Jens Axboe To: linux-kernel@vger.kernel.org Cc: peterz@infradead.org, tglx@linutronix.de, Jens Axboe Subject: [PATCH 2/4] sched/core: change rq->nr_iowait type to atomic_long_t Date: Mon, 19 Aug 2024 09:39:47 -0600 Message-ID: <20240819154259.215504-3-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240819154259.215504-1-axboe@kernel.dk> References: <20240819154259.215504-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In preparation for storing two separate iowait states in there, bump the size from a 32-bit to a 64-bit size, for 64-bit kernels. On 32-bit archs, the type remains a 32-bit size. Note that on 32-bit, the number of tasks are limited to 0x8000, which fits just fine in even half of the existing 32-bit atomic_t. For 64-bit, no such limit exists, hence play it safe and make it a 64-bit atomic. Signed-off-by: Jens Axboe Reviewed-by: Zhang Qiao --- kernel/sched/core.c | 8 ++++---- kernel/sched/sched.h | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 9bf1b67818d0..7e04b84dcc55 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3543,17 +3543,17 @@ static inline bool rq_has_pinned_tasks(struct rq *r= q) =20 static void task_iowait_inc(struct task_struct *p) { - atomic_inc(&task_rq(p)->nr_iowait); + atomic_long_inc(&task_rq(p)->nr_iowait); } =20 static void task_iowait_dec(struct task_struct *p) { - atomic_dec(&task_rq(p)->nr_iowait); + atomic_long_dec(&task_rq(p)->nr_iowait); } =20 int rq_iowait(struct rq *rq) { - return atomic_read(&rq->nr_iowait); + return atomic_long_read(&rq->nr_iowait); } =20 static void @@ -8372,7 +8372,7 @@ void __init sched_init(void) #endif #endif /* CONFIG_SMP */ hrtick_rq_init(rq); - atomic_set(&rq->nr_iowait, 0); + atomic_long_set(&rq->nr_iowait, 0); fair_server_init(rq); =20 #ifdef CONFIG_SCHED_CORE diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index b6b3b565bcb1..da2e67621f39 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1082,7 +1082,7 @@ struct rq { u64 clock_idle_copy; #endif =20 - atomic_t nr_iowait; + atomic_long_t nr_iowait; =20 #ifdef CONFIG_SCHED_DEBUG u64 last_seen_need_resched_ns; --=20 2.43.0 From nobody Sat Feb 7 18:15:41 2026 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2A68485956 for ; Mon, 19 Aug 2024 15:43:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724082191; cv=none; b=Kxb8BsKtuXzFNoyciGZS8noNJmwEyg3IiJkCw5iewbuv9b35SGsxMG7iSNVzG+2MndtNAL43V/aLzp320Wp0vLWf4XPLbY3pWkiXm+sOlyaX/Cuo4NFnpTOAdrBbFkdgieHERDCSzfm0H6zkF3PxkJV83qf643/fkU5X5nX7kxI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724082191; c=relaxed/simple; bh=xpaeZrdYp7o/3H939Bd1V1rBqFcrF4MCrxgKeTiRIas=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YinSRReczgLpcH5n2GFJ1dxSfXgJEVyWvtYSla9g7Kh96r/z8PbLv9nq8Wshe+ASOHovUv5vZsd/YFQaLvc8gAWZh3BjcZ89tvrLQS0I8quD5tRl0K81eOdpLYV55Mp/+ObXl8j6hGYXeQk7B0azPQhquNfDX4YNEQoFOn7Wwgc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=N0lfgQIR; arc=none smtp.client-ip=209.85.216.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="N0lfgQIR" Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-2d3badec49eso606406a91.1 for ; Mon, 19 Aug 2024 08:43:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1724082188; x=1724686988; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ez+V6hFdUaTIjdgjh9Zv2YX3S3+fART/LHqREz/LfwM=; b=N0lfgQIRFx6f32LmlEu7Hx+ei4iSn4U85fYmy0XLF3/85I73voL8+eErFkf4YHvsy+ ddlGgo7gqhg/9xRfVgGojYYpueywR1X1MonZN4Hom6LPhpHC5g9Xh2kP3xVJaY/pcdvN 92Qy4mFPJZcXz7mMtue0AUz3Ft8Ez9Sxe/L/1+lqWFD3AvXvtnOMtBLYDRK3dXUrLXeR HmI8egdBxjo4XMeUAPgB3WqRxUdBR63gJ34mDTSpaKH/iMzFFGi/KKBSQLAqEgocgXzF KzFtOpYKh1//q7OUPKn+q+mrmvhOfGFGE39B7HwPiomRqt/mMljMBlIHBDhQljZ30BJ7 KKXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724082188; x=1724686988; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ez+V6hFdUaTIjdgjh9Zv2YX3S3+fART/LHqREz/LfwM=; b=XlmIZrLg+ShtZPqm+m42v5vEwslOFeF3VyJXYdtiJv83Q3VKNdZ6QRJJZGFwZjMhrB NjKQkbL3e9jHSv8PYKEXqSrP8R0YmqY8Mbue1Cx6OJHxXgyTMEbyjkxddCBzXCaTYabn YmCVJSJlWoC3a9a9xgQAl4MIBylsO2w0kwWh5MAAAreYeMZ67TaHLGjUpanNN2K2t7Kx 4ghCn9YoGw7BX86/17h6MKmjqHbR13DjMOrsPD7op39gepuc8lYx1XmeDH26sSJGNvk7 iiGMxi9j0bMDITAqK8XnZyNLORpif76jyIVtScKA78Lf6JcQpOpGMUNEUnknblu9/Pob xwrg== X-Gm-Message-State: AOJu0YzEKtkG9o/4gBURzH47fWCMIGhnKmoze+F7REOXl4oaCG7SWuHa qiafzAyWaJJfU7iSrsy19VJ73Otzy/T5Vrn5G/1q43XQGwE1llsVf2Nim1aEpJCTflanIVlXfC5 q X-Google-Smtp-Source: AGHT+IHzSilzhTDi90d4es4HlvKwZl+ChIAk+D+HDSxeVbBgQX93kpvpg31Y7NnyqaQpbCsWJr3t3g== X-Received: by 2002:a17:90a:2f42:b0:2cb:4382:99eb with SMTP id 98e67ed59e1d1-2d3e1736fffmr7535753a91.6.1724082187874; Mon, 19 Aug 2024 08:43:07 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d3e3c751f1sm7356041a91.38.2024.08.19.08.43.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Aug 2024 08:43:06 -0700 (PDT) From: Jens Axboe To: linux-kernel@vger.kernel.org Cc: peterz@infradead.org, tglx@linutronix.de, Jens Axboe Subject: [PATCH 3/4] sched/core: have io_schedule_prepare() return a long Date: Mon, 19 Aug 2024 09:39:48 -0600 Message-ID: <20240819154259.215504-4-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240819154259.215504-1-axboe@kernel.dk> References: <20240819154259.215504-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In preparation for needing more state then 32-bit on 64-bit archs, switch it to a long instead. Signed-off-by: Jens Axboe --- block/blk-cgroup.c | 2 +- include/linux/sched.h | 4 ++-- kernel/locking/mutex.c | 4 ++-- kernel/locking/rtmutex_api.c | 4 ++-- kernel/sched/core.c | 6 +++--- 5 files changed, 10 insertions(+), 10 deletions(-) diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index 69e70964398c..f8e6220c66a7 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -1884,7 +1884,7 @@ static void blkcg_maybe_throttle_blkg(struct blkcg_gq= *blkg, bool use_memdelay) u64 now =3D blk_time_get_ns(); u64 exp; u64 delay_nsec =3D 0; - int tok; + long tok; =20 while (blkg->parent) { int use_delay =3D atomic_read(&blkg->use_delay); diff --git a/include/linux/sched.h b/include/linux/sched.h index 2c1b4ee3234f..c1a65e19a3ac 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -319,8 +319,8 @@ asmlinkage void preempt_schedule_irq(void); extern void schedule_rtlock(void); #endif =20 -extern int __must_check io_schedule_prepare(void); -extern void io_schedule_finish(int token); +extern long __must_check io_schedule_prepare(void); +extern void io_schedule_finish(long token); extern long io_schedule_timeout(long timeout); extern void io_schedule(void); =20 diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index cbae8c0b89ab..4a86ea6c7f19 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -830,7 +830,7 @@ EXPORT_SYMBOL_GPL(mutex_lock_interruptible_nested); void __sched mutex_lock_io_nested(struct mutex *lock, unsigned int subclass) { - int token; + long token; =20 might_sleep(); =20 @@ -1026,7 +1026,7 @@ EXPORT_SYMBOL(mutex_lock_killable); */ void __sched mutex_lock_io(struct mutex *lock) { - int token; + long token; =20 token =3D io_schedule_prepare(); mutex_lock(lock); diff --git a/kernel/locking/rtmutex_api.c b/kernel/locking/rtmutex_api.c index a6974d044593..ddf7f7f3f0b5 100644 --- a/kernel/locking/rtmutex_api.c +++ b/kernel/locking/rtmutex_api.c @@ -547,7 +547,7 @@ EXPORT_SYMBOL_GPL(mutex_lock_killable_nested); =20 void __sched mutex_lock_io_nested(struct mutex *lock, unsigned int subclas= s) { - int token; + long token; =20 might_sleep(); =20 @@ -579,7 +579,7 @@ EXPORT_SYMBOL(mutex_lock_killable); =20 void __sched mutex_lock_io(struct mutex *lock) { - int token =3D io_schedule_prepare(); + long token =3D io_schedule_prepare(); =20 __mutex_lock_common(lock, TASK_UNINTERRUPTIBLE, 0, NULL, _RET_IP_); io_schedule_finish(token); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 7e04b84dcc55..d6c494e33206 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -7415,16 +7415,16 @@ static inline void preempt_dynamic_init(void) { } =20 #endif /* CONFIG_PREEMPT_DYNAMIC */ =20 -int io_schedule_prepare(void) +long io_schedule_prepare(void) { - int old_iowait =3D current->in_iowait; + long old_iowait =3D current->in_iowait; =20 current->in_iowait =3D 1; blk_flush_plug(current->plug, true); return old_iowait; } =20 -void io_schedule_finish(int token) +void io_schedule_finish(long token) { current->in_iowait =3D token; } --=20 2.43.0 From nobody Sat Feb 7 18:15:41 2026 Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B0C113B287 for ; Mon, 19 Aug 2024 15:43:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724082192; cv=none; b=uH2ig0FruRzcKMpsRpClZ6r+Y8f9G0WdiG6QiP1d5krKjq5M5lz9AJG9gshLEsPUpr5u9cX56sZdFxa9QXt5yyXcjU70GaXn6lBLIAzfUdStvgB6eZlBmNj7BNI8prWxn8KND4zhmKg3p1nUjW0Xc1YTcEeunU40R2kkhPlho1o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724082192; c=relaxed/simple; bh=TR7xaH6L9jgxRR/GiRLfQKqo83/pPQC7QNSPfZayCzw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ocn6bfAbl5fu11KiMWvJRjtp/1Qek6ulEEeoN0KDb33qFtRjHJUDlGZUp9d1zKP1J+crA/JNysdoJrBzKVXCEyFDkwFC+zjETUHTjDADaVa5OGC5MNwpNSZVRKdG/h5aVgbgKw3Evv3HIpWcnfxxzFvyM5rFgWiEglEim9um0fk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=OgzqL1b+; arc=none smtp.client-ip=209.85.216.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="OgzqL1b+" Received: by mail-pj1-f51.google.com with SMTP id 98e67ed59e1d1-2d3cc6170eeso802282a91.1 for ; Mon, 19 Aug 2024 08:43:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1724082189; x=1724686989; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SrNxaKdN0nj20MhEnsw6+1Kaiw7B64qE6Xvz9cx2Ues=; b=OgzqL1b+qrBAdYd9KV7j3OvQFvN7s7cTEqwdy9io/yiACumhoLlZZNsRb+5rWZLS7A 4uuTBJoDojTfw20QYwL7SZq9BOL3dBvSRxhRd7juc0lF4zL2Xiv7Rq86LmRmyh9j2C3U qr0DXEgz69sb3n7sj4KR2sIaPjAMrFy5FRwoOM2tYLATOGSP7ikGRsqd2GceRxpWQXKO Xtr7f169hK7ABOPJdaG7M3y+IOCsN2OjPX3XgCzj6W+kKCtSJGUigCLz+usi+qtDl76f 10Vn57WrP0mYGVrksHacIGJHsrRRJsGyvX04P2aKVTCZBe4WeV4VMtImZwcIox014ob7 6VzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724082189; x=1724686989; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SrNxaKdN0nj20MhEnsw6+1Kaiw7B64qE6Xvz9cx2Ues=; b=fj0CD6Jqmr82Ejso7edyt7t5VX6TI4GpRzfl5ezlYwUpYj4QIebFJ+IkSO7OxvlEUx Gt/2TUMYdV4EU0QSzPmD0AKgeIV634hk80VhDe7M3pxaJr+7adFLzcOVShv2bl5QkTh7 hnssTSNKLBAiGKow8fJGEY9HTLL3xT9RYTmA+WwFOXQFsYye8iAauvnGZBGMdoEpQm4S fLeeJHHRUXXXzpr+Tyw47rl8jjRrejU9oMkm8YHlAxh/VAEnYEI9uaaJH8rPl+lz8vvc d/WHLTyZQnLCQdmWJOcbi/adFt/141MijM0uTwgt3eAZF2ATRGIWRzW9RQugCXI2bwUR Wqlw== X-Gm-Message-State: AOJu0YxftbLkfgfguF0dz/V+vjBUG+4fyQg7D6jFltAaWTuXMvHJHVDA rj9tZw+henDzDpxhpCaw0Eb4lDx8YWCQpTrApQ0APF22qhCNXRlf2pX68uLhThB0CFzk3BtwD5J E X-Google-Smtp-Source: AGHT+IF4gcFLJSSpe+tu9+lz3xS5xKspsgXRZSsGGQ7GI6UPMwVEfS71qppmaH8I0RuzWx2ZKRb8jA== X-Received: by 2002:a17:90a:c390:b0:2d3:d662:61ec with SMTP id 98e67ed59e1d1-2d3e04140f2mr7114441a91.4.1724082189427; Mon, 19 Aug 2024 08:43:09 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d3e3c751f1sm7356041a91.38.2024.08.19.08.43.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Aug 2024 08:43:08 -0700 (PDT) From: Jens Axboe To: linux-kernel@vger.kernel.org Cc: peterz@infradead.org, tglx@linutronix.de, Jens Axboe Subject: [PATCH 4/4] sched/core: split iowait state into two states Date: Mon, 19 Aug 2024 09:39:49 -0600 Message-ID: <20240819154259.215504-5-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240819154259.215504-1-axboe@kernel.dk> References: <20240819154259.215504-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" iowait is a bogus metric, but it's helpful in the sense that it allows short waits to not enter sleep states that have a higher exit latency than would've otherwise have been picked for iowait'ing tasks. However, it's harmless in that lots of applications and monitoring assumes that iowait is busy time, or otherwise use it as a health metric. Particularly for async IO it's entirely nonsensical. Split the iowait part into two parts - one that tracks whether the task needs boosting for short waits, and one that marks whether the task needs to be accounted as such. ->in_iowait_acct nests inside of ->in_iowait, both for efficiency reasons, but also so that the relationship between the two is clear. A waiter may set ->in_wait alone and not care about the accounting. Existing users of nr_iowait() for accounting purposes are switched to use nr_iowait_acct(), which leaves the governor using nr_iowait() as it only cares about iowaiters, not the accounting side. Utilize that there's enough space in rq->nr_iowait to store both values in there, shifting the accounting side by half the size of the type. Thank you to Thomas Gleixner for that [1] suggestion. [1] https://lore.kernel.org/lkml/87sf1b6o9w.ffs@tglx/ Signed-off-by: Jens Axboe --- arch/s390/appldata/appldata_base.c | 2 +- arch/s390/appldata/appldata_os.c | 2 +- fs/proc/stat.c | 2 +- include/linux/sched.h | 6 ++++ include/linux/sched/stat.h | 5 ++-- kernel/sched/core.c | 45 +++++++++++++++++++++++------- kernel/sched/sched.h | 1 + kernel/time/tick-sched.c | 6 ++-- 8 files changed, 51 insertions(+), 18 deletions(-) diff --git a/arch/s390/appldata/appldata_base.c b/arch/s390/appldata/applda= ta_base.c index 91a30e017d65..5b21225405fd 100644 --- a/arch/s390/appldata/appldata_base.c +++ b/arch/s390/appldata/appldata_base.c @@ -423,4 +423,4 @@ EXPORT_SYMBOL_GPL(si_swapinfo); #endif EXPORT_SYMBOL_GPL(nr_threads); EXPORT_SYMBOL_GPL(nr_running); -EXPORT_SYMBOL_GPL(nr_iowait); +EXPORT_SYMBOL_GPL(nr_iowait_acct); diff --git a/arch/s390/appldata/appldata_os.c b/arch/s390/appldata/appldata= _os.c index a363d30ce739..fa4b278aca6c 100644 --- a/arch/s390/appldata/appldata_os.c +++ b/arch/s390/appldata/appldata_os.c @@ -100,7 +100,7 @@ static void appldata_get_os_data(void *data) =20 os_data->nr_threads =3D nr_threads; os_data->nr_running =3D nr_running(); - os_data->nr_iowait =3D nr_iowait(); + os_data->nr_iowait =3D nr_iowait_acct(); os_data->avenrun[0] =3D avenrun[0] + (FIXED_1/200); os_data->avenrun[1] =3D avenrun[1] + (FIXED_1/200); os_data->avenrun[2] =3D avenrun[2] + (FIXED_1/200); diff --git a/fs/proc/stat.c b/fs/proc/stat.c index da60956b2915..149be7a884fb 100644 --- a/fs/proc/stat.c +++ b/fs/proc/stat.c @@ -180,7 +180,7 @@ static int show_stat(struct seq_file *p, void *v) (unsigned long long)boottime.tv_sec, total_forks, nr_running(), - nr_iowait()); + nr_iowait_acct()); =20 seq_put_decimal_ull(p, "softirq ", (unsigned long long)sum_softirq); =20 diff --git a/include/linux/sched.h b/include/linux/sched.h index c1a65e19a3ac..1b40847b0a80 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -953,7 +953,13 @@ struct task_struct { =20 /* Bit to tell TOMOYO we're in execve(): */ unsigned in_execve:1; + /* task is in iowait */ unsigned in_iowait:1; + /* + * task is in iowait and should be accounted as such. can only be set + * if ->in_iowait is also set. + */ + unsigned in_iowait_acct:1; #ifndef TIF_RESTORE_SIGMASK unsigned restore_sigmask:1; #endif diff --git a/include/linux/sched/stat.h b/include/linux/sched/stat.h index 0108a38bb64d..31e8a44b3d71 100644 --- a/include/linux/sched/stat.h +++ b/include/linux/sched/stat.h @@ -19,8 +19,9 @@ DECLARE_PER_CPU(unsigned long, process_counts); extern int nr_processes(void); extern unsigned int nr_running(void); extern bool single_task_running(void); -extern unsigned int nr_iowait(void); -extern unsigned int nr_iowait_cpu(int cpu); +unsigned int nr_iowait_acct(void); +unsigned int nr_iowait_acct_cpu(int cpu); +unsigned int nr_iowait_cpu(int cpu); =20 static inline int sched_info_on(void) { diff --git a/kernel/sched/core.c b/kernel/sched/core.c index d6c494e33206..a082919caaf8 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3541,19 +3541,33 @@ static inline bool rq_has_pinned_tasks(struct rq *r= q) =20 #endif /* !CONFIG_SMP */ =20 +/* Shift half the size of the type, atomic_long_t */ +#define IOWAIT_SHIFT (4 * sizeof(atomic_long_t)) + +/* + * Store iowait and iowait_acct state in the same variable. The lower bits + * hold the iowait state, and the upper bits hold the iowait_acct state. + */ static void task_iowait_inc(struct task_struct *p) { - atomic_long_inc(&task_rq(p)->nr_iowait); + long val =3D 1 + ((long) p->in_iowait_acct << IOWAIT_SHIFT); + atomic_long_add(val, &task_rq(p)->nr_iowait); } =20 static void task_iowait_dec(struct task_struct *p) { - atomic_long_dec(&task_rq(p)->nr_iowait); + long val =3D 1 + ((long) p->in_iowait_acct << IOWAIT_SHIFT); + atomic_long_sub(val, &task_rq(p)->nr_iowait); } =20 int rq_iowait(struct rq *rq) { - return atomic_long_read(&rq->nr_iowait); + return atomic_long_read(&rq->nr_iowait) & ((1UL << IOWAIT_SHIFT) - 1); +} + +int rq_iowait_acct(struct rq *rq) +{ + return atomic_long_read(&rq->nr_iowait) >> IOWAIT_SHIFT; } =20 static void @@ -5295,7 +5309,12 @@ unsigned long long nr_context_switches(void) * it does become runnable. */ =20 -unsigned int nr_iowait_cpu(int cpu) +unsigned int nr_iowait_acct_cpu(int cpu) +{ + return rq_iowait_acct(cpu_rq(cpu)); +} + +unsigned nr_iowait_cpu(int cpu) { return rq_iowait(cpu_rq(cpu)); } @@ -5330,12 +5349,12 @@ unsigned int nr_iowait_cpu(int cpu) * Task CPU affinities can make all that even more 'interesting'. */ =20 -unsigned int nr_iowait(void) +unsigned int nr_iowait_acct(void) { unsigned int i, sum =3D 0; =20 for_each_possible_cpu(i) - sum +=3D nr_iowait_cpu(i); + sum +=3D nr_iowait_acct_cpu(i); =20 return sum; } @@ -7415,18 +7434,24 @@ static inline void preempt_dynamic_init(void) { } =20 #endif /* CONFIG_PREEMPT_DYNAMIC */ =20 +/* + * Returns a token which is comprised of the two bits of iowait wait state= - + * one is whether we're making ourselves as in iowait for cpufreq reasons, + * and the other is if the task should be accounted as such. + */ long io_schedule_prepare(void) { - long old_iowait =3D current->in_iowait; - + long token =3D current->in_iowait + ((long) current->in_iowait_acct << IO= WAIT_SHIFT); current->in_iowait =3D 1; + current->in_iowait_acct =3D 1; blk_flush_plug(current->plug, true); - return old_iowait; + return token; } =20 void io_schedule_finish(long token) { - current->in_iowait =3D token; + current->in_iowait =3D token & 0x01; + current->in_iowait_acct =3D token >> IOWAIT_SHIFT; } =20 /* diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index da2e67621f39..be377ecd1301 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -3580,6 +3580,7 @@ extern u64 avg_vruntime(struct cfs_rq *cfs_rq); extern int entity_eligible(struct cfs_rq *cfs_rq, struct sched_entity *se); =20 int rq_iowait(struct rq *rq); +int rq_iowait_acct(struct rq *rq); =20 #ifdef CONFIG_RT_MUTEXES =20 diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 753a184c7090..17a82771c3df 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -732,7 +732,7 @@ static void tick_nohz_stop_idle(struct tick_sched *ts, = ktime_t now) delta =3D ktime_sub(now, ts->idle_entrytime); =20 write_seqcount_begin(&ts->idle_sleeptime_seq); - if (nr_iowait_cpu(smp_processor_id()) > 0) + if (nr_iowait_acct_cpu(smp_processor_id()) > 0) ts->iowait_sleeptime =3D ktime_add(ts->iowait_sleeptime, delta); else ts->idle_sleeptime =3D ktime_add(ts->idle_sleeptime, delta); @@ -805,7 +805,7 @@ u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time) struct tick_sched *ts =3D &per_cpu(tick_cpu_sched, cpu); =20 return get_cpu_sleep_time_us(ts, &ts->idle_sleeptime, - !nr_iowait_cpu(cpu), last_update_time); + !nr_iowait_acct_cpu(cpu), last_update_time); } EXPORT_SYMBOL_GPL(get_cpu_idle_time_us); =20 @@ -831,7 +831,7 @@ u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_ti= me) struct tick_sched *ts =3D &per_cpu(tick_cpu_sched, cpu); =20 return get_cpu_sleep_time_us(ts, &ts->iowait_sleeptime, - nr_iowait_cpu(cpu), last_update_time); + nr_iowait_acct_cpu(cpu), last_update_time); } EXPORT_SYMBOL_GPL(get_cpu_iowait_time_us); =20 --=20 2.43.0