From nobody Mon Feb 9 17:56:46 2026 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E09EA83CCD for ; Tue, 16 Apr 2024 12:15:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713269737; cv=none; b=hlW3xng+OuBUeGScrGiXh7gjzkMpuIBJVGki/6kaEL6oH2kiGWqO+sRgmEow8pYsOv7mVkMxDoFnAwbpSV94w5gH7N/ySyxXuFfcpSX7mM3YWhI5XnM5tiEm4x0BM0rNqDTrC7ZVNvtNEmurlhVXEABi/aLK00g2Nx+4kypJeFs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713269737; c=relaxed/simple; bh=wwXZY5GdG4n0jEoi2bs4ssdvrV+7cuMAHqMUWo3Msys=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZBbBPQrbg/NmnHTt0g22DovILHSVluhNaWYw8Sq6faSqle+pCWzRu8uzTuFdgoDKHuv+L3KTWFNDuTli8UEWxc/TZ2TE68tUlLVnN+SdWnImDmXTkZuKepEDJR6chour55KRxvWO4/+hPFsO7HQ6Kc6MUutapEt0A4Nea5wGJoM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=Kjqs/xoo; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="Kjqs/xoo" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-1e2be2361efso7338335ad.0 for ; Tue, 16 Apr 2024 05:15:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1713269734; x=1713874534; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6zQPNoqoD7IacLS4osEkWLaaMBgy95aK11WInzHtzqI=; b=Kjqs/xooJStd001TbSMcIn8qYkZAsjXAC34YTO34iyiWicYWcWYFALZdQC5ED5gfsD iZ3cCk/XCpr6dYJ6cSsHPBf2prhhnywKU+e+RR37nWiLIABLCts6+/KpGqZeu9V8F+yE yQtsIiJwbogOnL8VE/IgklbE5NTpzbJ05G0HZ0u99h0vIPZO5acJpMhYNF9bjytqCctn ZUKD7KuqMbGwUxbzQo+thont3frc3LAGTUIWV4/XBVNyJ9m9sBCw0Ht107rfmqynAr+O 0barLJhXyPHVAgUGQZ0b8qqWtKnepe4jhGa6c7I777iMdldGlNAIJSiX/2Jqt0Cxj9OR LgDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713269734; x=1713874534; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6zQPNoqoD7IacLS4osEkWLaaMBgy95aK11WInzHtzqI=; b=Jtj84IfwKq70nA9cgycqGWLDxXR+ASYGV3x4uf+mAx5kzwmnGi8cHYXpoWZV7VBhVu epz04H222U6b+9VY6QDSITYifo16MEoDLGgpzpFEEU65L4+uk+aNbwO6khs6oNwiL311 kGpq+b9dPl597VY3XB8gw9ldKs92Cjo5P5rTBVsyQj0jnZHkCdGpsOL/SprgKBMeC/dG 4/o4WIW8s5GmkgRfjrbE4xImAzkmha8p4BsrpYPwa/+5yBhzrSM4qHQK8Ng2GsWRgUon wQjQS8nnxIbqBXPTApmbqanHX3s6KCt2CphXB9OEEhsikHkBKfTCSrxCuytNu+3K7Q8x oqEw== X-Gm-Message-State: AOJu0Yy+fV/17jRWGMtJ4O6eRyI6MMkx4m/nJnze8tHJn0sLtlfrtuiV BK1Gfv2o9D0xgwx0ak+zdQ8L29t8H8e23nAwmJgGFM7hvPoEFZOHQLmgY7XLogPDYfGWNf5YJ0J F X-Google-Smtp-Source: AGHT+IFIO+91FCVJHbHnNWsAVEM+LOCE4SnQPDZzAR6MaodLTB0LVa0OognIPTOyoRIvkicclF38QA== X-Received: by 2002:a17:902:f542:b0:1e2:b3d:8c67 with SMTP id h2-20020a170902f54200b001e20b3d8c67mr14774923plf.6.1713269734176; Tue, 16 Apr 2024 05:15:34 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id d12-20020a170903230c00b001e43576a7a1sm9584693plh.222.2024.04.16.05.15.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Apr 2024 05:15:32 -0700 (PDT) From: Jens Axboe To: linux-kernel@vger.kernel.org Cc: peterz@infradead.org, tglx@linutronix.de, Jens Axboe Subject: [PATCH 1/4] sched/core: add helpers for iowait handling Date: Tue, 16 Apr 2024 06:11:18 -0600 Message-ID: <20240416121526.67022-2-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240416121526.67022-1-axboe@kernel.dk> References: <20240416121526.67022-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Adds helpers to inc/dec the runqueue iowait count, based on the task, and use those in the spots where the count is manipulated. Adds an rq_iowait() helper, to abstract out how the per-rq stats are read. No functional changes in this patch, just in preparation for switching the type of 'nr_iowait'. Signed-off-by: Jens Axboe --- kernel/sched/core.c | 23 +++++++++++++++++++---- kernel/sched/cputime.c | 3 +-- kernel/sched/sched.h | 2 ++ 3 files changed, 22 insertions(+), 6 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 7019a40457a6..977bb08a33d2 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3721,6 +3721,21 @@ static inline cpumask_t *alloc_user_cpus_ptr(int nod= e) =20 #endif /* !CONFIG_SMP */ =20 +static void task_iowait_inc(struct task_struct *p) +{ + atomic_inc(&task_rq(p)->nr_iowait); +} + +static void task_iowait_dec(struct task_struct *p) +{ + atomic_dec(&task_rq(p)->nr_iowait); +} + +int rq_iowait(struct rq *rq) +{ + return atomic_read(&rq->nr_iowait); +} + static void ttwu_stat(struct task_struct *p, int cpu, int wake_flags) { @@ -3787,7 +3802,7 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p= , int wake_flags, #endif if (p->in_iowait) { delayacct_blkio_end(p); - atomic_dec(&task_rq(p)->nr_iowait); + task_iowait_dec(p); } =20 activate_task(rq, p, en_flags); @@ -4364,7 +4379,7 @@ int try_to_wake_up(struct task_struct *p, unsigned in= t state, int wake_flags) if (task_cpu(p) !=3D cpu) { if (p->in_iowait) { delayacct_blkio_end(p); - atomic_dec(&task_rq(p)->nr_iowait); + task_iowait_dec(p); } =20 wake_flags |=3D WF_MIGRATED; @@ -5472,7 +5487,7 @@ unsigned long long nr_context_switches(void) =20 unsigned int nr_iowait_cpu(int cpu) { - return atomic_read(&cpu_rq(cpu)->nr_iowait); + return rq_iowait(cpu_rq(cpu)); } =20 /* @@ -6692,7 +6707,7 @@ static void __sched notrace __schedule(unsigned int s= ched_mode) deactivate_task(rq, prev, DEQUEUE_SLEEP | DEQUEUE_NOCLOCK); =20 if (prev->in_iowait) { - atomic_inc(&rq->nr_iowait); + task_iowait_inc(prev); delayacct_blkio_start(); } } diff --git a/kernel/sched/cputime.c b/kernel/sched/cputime.c index af7952f12e6c..7d9423df7779 100644 --- a/kernel/sched/cputime.c +++ b/kernel/sched/cputime.c @@ -222,9 +222,8 @@ void account_steal_time(u64 cputime) void account_idle_time(u64 cputime) { u64 *cpustat =3D kcpustat_this_cpu->cpustat; - struct rq *rq =3D this_rq(); =20 - if (atomic_read(&rq->nr_iowait) > 0) + if (rq_iowait(this_rq()) > 0) cpustat[CPUTIME_IOWAIT] +=3D cputime; else cpustat[CPUTIME_IDLE] +=3D cputime; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index d2242679239e..387f67ddf18a 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -3473,4 +3473,6 @@ static inline void init_sched_mm_cid(struct task_stru= ct *t) { } extern u64 avg_vruntime(struct cfs_rq *cfs_rq); extern int entity_eligible(struct cfs_rq *cfs_rq, struct sched_entity *se); =20 +int rq_iowait(struct rq *rq); + #endif /* _KERNEL_SCHED_SCHED_H */ --=20 2.43.0 From nobody Mon Feb 9 17:56:46 2026 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 610C312AAE2 for ; Tue, 16 Apr 2024 12:15:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713269739; cv=none; b=I8/ZjHI77c2Lh/2mRwX42F9OKy/SZ8mm1K9m7xX9wmfWABEw04c9QVNMFpzrZzMqjkjWq5Q/GbGXFRoyo8el6E03xLVN1WqCm720T0UVxQHoh1qzZjs53SsSn+IN8m/8EFdl3k6WwWFqxVYKiBb1r+vNly7PXLPITcayndBwCrk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713269739; c=relaxed/simple; bh=LFTwl+KaNR01xUhsIjDuW6dm42jh/bpLqCuATv5koSk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=I0BXcjcNwmP96KuJei/da57Fo890Au135qVqxcTKdpui1lVcR0oUD6nmBaCl0+3e5MXq6H5u5DMj+37Sna2433It3t/fQPbZxcKqWLqMouWVNgA+9vHY0BOwqhxw60GIUYrwjySL0rnWmefeAtnBtejMBMQt8KLXIwww88Eq4uI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=AkAkzVLp; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="AkAkzVLp" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-6f013c304bbso377151b3a.1 for ; Tue, 16 Apr 2024 05:15:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1713269736; x=1713874536; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=M8FZtLez5NK90qpzb7pJiE/cmJLi/g0rDQRNg4yxT3c=; b=AkAkzVLpXD7vhIZlLh9bC9VAHIHGZf5OSN8xvZhdYjiuYE5WWs0OEv9ZnbKASq7+dp aUA/u0VxKpJBUMOnbVPVtVsikuTPc4VVd+v7I6lr77Mxz8C2Dn5X7AgNRxGS08ivFNbB ubOSIG2Fg0d3gHblVS5WwMUqTvbA5qJBDUSwexlVlWWhSKzcnGH21OGx90RAM7QwM6Do W0cVAkM/EhAMT9BaCB9cjUFhD/bh3NloKvTNBSwBWPxPTkxFKSh1xM/9Q3uKMzGkLmlj nRc8o8Z5QNbqLbCDPrMehe38mpqKU8WuVlSyCSBae/WgRo6cBpS+Weos0KbUdBJEAOwo uA/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713269736; x=1713874536; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=M8FZtLez5NK90qpzb7pJiE/cmJLi/g0rDQRNg4yxT3c=; b=n4iFyByZYn3GmIrVpjrr18LjWVXv1e6liAb0rg0uxjJUJlYuQsEQnsQKsT9vmIu9vp h5mojnGXN7/zOqTNuP2PoJwubOQ2u+c0A+qyMI+eP/o7PzIgoRe7xJ5DGjZdo8jSLQ0h zYztH4mq7ONyH0u7sPkKqL9QzCaR6NxSEOdR6BbgciR7hiUDQqF7lWjLf0mFDs43hO9I rfHj8i+If30wKLMGH6HbGD8UsWwHHmcIS6hHAsZSR7z8PHIe4xfNuZ4/yD3Qq07B3dlV qXZQCein4kPHA2XUL7NCyMecWFSbRgArkRFqnDbdIL2Vso/l21FNktxKdLXgx219ilDc 0jaA== X-Gm-Message-State: AOJu0Yy6dTjMlAWYkn6w8sfbU5VvVei9b7MrwFATTckM87vN1sQVQl0N WjqT5o54WTMz2hntZ3JStkHFaAlgBVou+dWi+JuOpYihAx6Y6tKnKviPLREo+j4RoMWQccKS/bc o X-Google-Smtp-Source: AGHT+IGFn1ukMhmFaOaV3SngWL/VAPF/owOpZcgy22GXhm7nBqiZcEQ4Emt0z452eARBAlN6Qn3iVw== X-Received: by 2002:a17:902:ce8d:b0:1e4:344e:768e with SMTP id f13-20020a170902ce8d00b001e4344e768emr15013803plg.5.1713269736213; Tue, 16 Apr 2024 05:15:36 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id d12-20020a170903230c00b001e43576a7a1sm9584693plh.222.2024.04.16.05.15.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Apr 2024 05:15:34 -0700 (PDT) From: Jens Axboe To: linux-kernel@vger.kernel.org Cc: peterz@infradead.org, tglx@linutronix.de, Jens Axboe Subject: [PATCH 2/4] sched/core: change rq->nr_iowait type to atomic_long_t on 64-bit Date: Tue, 16 Apr 2024 06:11:19 -0600 Message-ID: <20240416121526.67022-3-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240416121526.67022-1-axboe@kernel.dk> References: <20240416121526.67022-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In preparation for storing two separate iowait states in there, bump the size from a 32-bit to a 64-bit size, for 64-bit kernels. Note that on 32-bit, the number of tasks are limited to 0x8000, which fits just fine in even half of the existiing 32-bit atomic_t. For 64-bit, no such limit exists, hence play it safe and make it a 64-bit atomic. Signed-off-by: Jens Axboe --- kernel/sched/core.c | 14 +++++++++++++- kernel/sched/sched.h | 4 ++++ 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 977bb08a33d2..6a6c985220b1 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3723,17 +3723,29 @@ static inline cpumask_t *alloc_user_cpus_ptr(int no= de) =20 static void task_iowait_inc(struct task_struct *p) { +#ifdef CONFIG_64BIT + atomic_long_inc(&task_rq(p)->nr_iowait); +#else atomic_inc(&task_rq(p)->nr_iowait); +#endif } =20 static void task_iowait_dec(struct task_struct *p) { +#ifdef CONFIG_64BIT + atomic_long_dec(&task_rq(p)->nr_iowait); +#else atomic_dec(&task_rq(p)->nr_iowait); +#endif } =20 int rq_iowait(struct rq *rq) { +#ifdef CONFIG_64BIT + return atomic_long_read(&rq->nr_iowait); +#else return atomic_read(&rq->nr_iowait); +#endif } =20 static void @@ -10065,7 +10077,7 @@ void __init sched_init(void) #endif #endif /* CONFIG_SMP */ hrtick_rq_init(rq); - atomic_set(&rq->nr_iowait, 0); + atomic_long_set(&rq->nr_iowait, 0); =20 #ifdef CONFIG_SCHED_CORE rq->core =3D rq; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 387f67ddf18a..c2802d066615 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1049,7 +1049,11 @@ struct rq { u64 clock_idle_copy; #endif =20 +#ifdef CONFIG_64BIT + atomic_long_t nr_iowait; +#else atomic_t nr_iowait; +#endif =20 #ifdef CONFIG_SCHED_DEBUG u64 last_seen_need_resched_ns; --=20 2.43.0 From nobody Mon Feb 9 17:56:46 2026 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1715F12BF29 for ; Tue, 16 Apr 2024 12:15:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713269742; cv=none; b=jdJyo733QkCfCRwB+f/qHw8ChSuwesKqprquSqMspWdLQs6y05btbXrOhmoGQ4iUUlj9d69LBBWJuXI1D4jwzVUZn/Qn51enkNkKAEM+03YrftHZNzQq7hj5uL7sW9l59IHgEtn8d/910nXMnU2/2KTjCAFcLyyI4/8DMhRXqkc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713269742; c=relaxed/simple; bh=RG9FrqpOf7WbjmJJz8QqVZnfuVsS9/NaC14Fu5Zb+ig=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JL2W9G2pO16WU1uBnI4AG4YGqIwtmXEl0IvYw3eixBbP30hCoHTO8bZB2xpOPkZWYLWCacTbMLQiMw/GtcMg/RZKO7JN5hztNiamVQ7s45z0ZIKgj6ttCvohboqP8Y2i08KEl9CA8yPKVrJySYG2+r/B4NoY8F1XBCEtbR/MkxM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=owORLR12; arc=none smtp.client-ip=209.85.215.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="owORLR12" Received: by mail-pg1-f170.google.com with SMTP id 41be03b00d2f7-5f0a485dab9so372451a12.1 for ; Tue, 16 Apr 2024 05:15:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1713269738; x=1713874538; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tn8L7Kj7xLJGqKx7jb5AwYrGu1BZjVWnoiTJPrrq0So=; b=owORLR12QMVsGgrsVx6FgVXKUof0kex0BiBBje+9s49yFD+aoPWbi2cDhDAhp1P8OH KIPlXKT9BI/dZWskUBTCATE42I5fDvq8u0sbjhlB3r9EHbrD836xfrabIWVAldH1x/re TCMxDWfhGawMR/XOZbFMrLrs4gz2xY3eFGoKY0UT4hOic8duXKKRuBpaoppc09RzbbrC +2kO8xZN4d2yqNhdjQEDR9K9drltQBOjOgx9MRp0ecJwK9ttw21Pc87Wob1s3kYb2GOF bB5zgBUSAPMwxLs08aBVBAoSIKVMHq+U5Hy2jq1i4IN+iXXNVCMB4qhEJZRLeJY5WRge Liag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713269738; x=1713874538; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tn8L7Kj7xLJGqKx7jb5AwYrGu1BZjVWnoiTJPrrq0So=; b=MkAUuZKO4UAyYPT3BbJjvTVJSiqn31Aja6hf+Jgbh7OdkRg4PpPHMXt0Z3WYIJZljI 9d5sgwknDLio3kwLCapiFPsk2vKJmx/t3/86h3kUGDsxhvdiW5k5doFVuL3VGioYHJ8t dXTaCy9CSduSv4W5aKh/G4GEZlLX4Mld6e4GJjhiUK+JL9+4X5mmHGruVuCdf5LWrAbS ir7R2PcxiBCQGgNhMgYpxroco1IEry7IPGWOLZ6vvcZBIFGgGjEL0ZrGevjeti8A27rY 3iWSMqJc1E4c8X/O0HUEy8WrCIv+pBsl7cvHDWdjka8SNGtBGmCsTLuaGJCbk8Xi0wwo vJTw== X-Gm-Message-State: AOJu0Yymy1ospmyoOxArVoDFkH4CnW8Q6od4OXeJDttPxHVd/2XSBslV /rru468pPvmIanErPh83e44e23XLIp52c8ZjhaHXEdf/vwhahMnndAHXS05T8MEmV0sIehP3xXc h X-Google-Smtp-Source: AGHT+IHxW6aW0PxXuhIyhTHTYRuVxyaLu2t34mwID/dWVGQtCCT5fRQR0E1e+inB2kl18UPNghhbnA== X-Received: by 2002:a17:902:e84d:b0:1dd:da28:e5ca with SMTP id t13-20020a170902e84d00b001ddda28e5camr14826585plg.0.1713269737731; Tue, 16 Apr 2024 05:15:37 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id d12-20020a170903230c00b001e43576a7a1sm9584693plh.222.2024.04.16.05.15.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Apr 2024 05:15:36 -0700 (PDT) From: Jens Axboe To: linux-kernel@vger.kernel.org Cc: peterz@infradead.org, tglx@linutronix.de, Jens Axboe Subject: [PATCH 3/4] sched/core: have io_schedule_prepare() return a long Date: Tue, 16 Apr 2024 06:11:20 -0600 Message-ID: <20240416121526.67022-4-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240416121526.67022-1-axboe@kernel.dk> References: <20240416121526.67022-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In preparation for needing more state then 32-bit on 64-bit archs, switch it to a long instead. Signed-off-by: Jens Axboe --- block/blk-cgroup.c | 2 +- include/linux/sched.h | 4 ++-- kernel/locking/mutex.c | 4 ++-- kernel/locking/rtmutex_api.c | 4 ++-- kernel/sched/core.c | 6 +++--- 5 files changed, 10 insertions(+), 10 deletions(-) diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index bdbb557feb5a..77faceddd5dd 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -1849,7 +1849,7 @@ static void blkcg_maybe_throttle_blkg(struct blkcg_gq= *blkg, bool use_memdelay) u64 now =3D blk_time_get_ns(); u64 exp; u64 delay_nsec =3D 0; - int tok; + long tok; =20 while (blkg->parent) { int use_delay =3D atomic_read(&blkg->use_delay); diff --git a/include/linux/sched.h b/include/linux/sched.h index 3c2abbc587b4..dcfc2830ed8e 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -317,8 +317,8 @@ asmlinkage void preempt_schedule_irq(void); extern void schedule_rtlock(void); #endif =20 -extern int __must_check io_schedule_prepare(void); -extern void io_schedule_finish(int token); +extern long __must_check io_schedule_prepare(void); +extern void io_schedule_finish(long token); extern long io_schedule_timeout(long timeout); extern void io_schedule(void); =20 diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index cbae8c0b89ab..4a86ea6c7f19 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -830,7 +830,7 @@ EXPORT_SYMBOL_GPL(mutex_lock_interruptible_nested); void __sched mutex_lock_io_nested(struct mutex *lock, unsigned int subclass) { - int token; + long token; =20 might_sleep(); =20 @@ -1026,7 +1026,7 @@ EXPORT_SYMBOL(mutex_lock_killable); */ void __sched mutex_lock_io(struct mutex *lock) { - int token; + long token; =20 token =3D io_schedule_prepare(); mutex_lock(lock); diff --git a/kernel/locking/rtmutex_api.c b/kernel/locking/rtmutex_api.c index a6974d044593..ddf7f7f3f0b5 100644 --- a/kernel/locking/rtmutex_api.c +++ b/kernel/locking/rtmutex_api.c @@ -547,7 +547,7 @@ EXPORT_SYMBOL_GPL(mutex_lock_killable_nested); =20 void __sched mutex_lock_io_nested(struct mutex *lock, unsigned int subclas= s) { - int token; + long token; =20 might_sleep(); =20 @@ -579,7 +579,7 @@ EXPORT_SYMBOL(mutex_lock_killable); =20 void __sched mutex_lock_io(struct mutex *lock) { - int token =3D io_schedule_prepare(); + long token =3D io_schedule_prepare(); =20 __mutex_lock_common(lock, TASK_UNINTERRUPTIBLE, 0, NULL, _RET_IP_); io_schedule_finish(token); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 6a6c985220b1..63f6d44f460c 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -9032,16 +9032,16 @@ int __sched yield_to(struct task_struct *p, bool pr= eempt) } EXPORT_SYMBOL_GPL(yield_to); =20 -int io_schedule_prepare(void) +long io_schedule_prepare(void) { - int old_iowait =3D current->in_iowait; + long old_iowait =3D current->in_iowait; =20 current->in_iowait =3D 1; blk_flush_plug(current->plug, true); return old_iowait; } =20 -void io_schedule_finish(int token) +void io_schedule_finish(long token) { current->in_iowait =3D token; } --=20 2.43.0 From nobody Mon Feb 9 17:56:46 2026 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3E8E512C477 for ; Tue, 16 Apr 2024 12:15:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713269744; cv=none; b=pi7JfqVIgAjF5Ds9biI5jKqsI1yTBFKFYMHcfN/Qv3zJE2pIPNzlzJrfWsaQOeWQgxkxtSSCNF1kBKWuuSiDM6gHCu+7wXNz0E4ae88QeO4jLmlyqw9zxcAtsWprhkxoN0sS1B6TajYpmfvclMrYARreuMSmKlP5JZH1zog8bpM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713269744; c=relaxed/simple; bh=VSoxAqTuDxa+6uoGvfJFwbhJiCuybB38cw3O7vNfl1s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KKI9IRdB2s5ZhtXtGu7Lp/Bqh90bX+tzZymaeT8NMnjSulE+n8ap2ZRIyagZ5bq0UhdtcRpslydj2RnW9C9DKDAuWbd0/vsDUcitj0woNko3C5Ius+BHMdktSUD5RBeGRBYnDx8fuIXDGj2mvAQ1Ogypn+MJYTKzfyy1psHsKSI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=ZGlE3qE8; arc=none smtp.client-ip=209.85.215.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="ZGlE3qE8" Received: by mail-pg1-f170.google.com with SMTP id 41be03b00d2f7-58962bf3f89so933204a12.0 for ; Tue, 16 Apr 2024 05:15:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1713269740; x=1713874540; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1PW6BtyfwXVpH9eVQnlmZoioSuL4oqjoVMV11U1Fnm0=; b=ZGlE3qE8OkajM7B2cUZS8s8xPjAcdl521EjEuGYemCcEBzh9nDATLG1AGzzvn/lIX6 7SyktK/nT6ET6xh3dz9lQpjvcDHK4Wn1tWodaYt4tvDrCpkfS29GJ1MlNgC6Fma/UP15 FbKr9EZmfbiYEQy/1+Kf9hwML0/mvEItKbII3B9XQIilDhxH047pqFNgPkHD1WIc0qPP wt4tAxH3QmijbYsxrhi2czlEtUq4ARsVHZY/PSmz0atfpQ8pJzvuuwTKpweKP94sEdO6 m3IVRUevPBydC3ACNcJ3Jnzi7Yf0p7209EIj3rZBSzGB195TnfGa1/6WW1u4H5TxuHLS ji6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713269740; x=1713874540; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1PW6BtyfwXVpH9eVQnlmZoioSuL4oqjoVMV11U1Fnm0=; b=v8eQaxQFcHv8PIYGSMdNpuVZZH5tTIii1DFQSc5pKTwYR2LgfJhxueKXKU5sume79d DPXsabrhBTjsfZ722MS5yG5vb4Af8ZRBpOh45E0mZNjw9pKQ9xkenF6dlC9LJz4t4rLh zioPpqSSsgm/xmBIJRe3UwDaYyuRFtONbbfoV9Csu71IRFTRfQE6R9GPDwkCTt1rrjq4 eCsJb3OkGvEsplCg3c0h1OxNMymu1hGD18FbCTAGDjmrKC8w0fWjQm6QKOMiHodTkhHY OnRBK+IQZtFvpHdgjZlIF3uybnTanM+B14H0xL/9W3Z/xVwrxDTwili/vHpeXRuw2dm3 XLtQ== X-Gm-Message-State: AOJu0YzM9jePXjE1Zy48UD/TGj+wqZnGI8lldKiOo9NPxyb2appNP7Bl egtzmkCbmOlqT8zXdb46s654+StEO4uN9gDnbH5gqq+xuKMaEem3GVq2kfy13tpfYwBJdLmsJTz 6 X-Google-Smtp-Source: AGHT+IFiEimvabQ9W1pWfZxF5RlCzxrarUbVzqrMzUi9GJmZzunHkqQZ/2qPhileRkKKn5R8kcmXpw== X-Received: by 2002:a05:6a20:3cac:b0:1a7:9b9a:757b with SMTP id b44-20020a056a203cac00b001a79b9a757bmr17077897pzj.1.1713269739851; Tue, 16 Apr 2024 05:15:39 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id d12-20020a170903230c00b001e43576a7a1sm9584693plh.222.2024.04.16.05.15.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Apr 2024 05:15:38 -0700 (PDT) From: Jens Axboe To: linux-kernel@vger.kernel.org Cc: peterz@infradead.org, tglx@linutronix.de, Jens Axboe Subject: [PATCH 4/4] sched/core: split iowait state into two states Date: Tue, 16 Apr 2024 06:11:21 -0600 Message-ID: <20240416121526.67022-5-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240416121526.67022-1-axboe@kernel.dk> References: <20240416121526.67022-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" iowait is a bogus metric, but it's helpful in the sense that it allows short waits to not enter sleep states that have a higher exit latency than would've otherwise have been picked for iowait'ing tasks. However, it's harmless in that lots of applications and monitoring assumes that iowait is busy time, or otherwise use it as a health metric. Particularly for async IO it's entirely nonsensical. Split the iowait part into two parts - one that tracks whether the task needs boosting for short waits, and one that marks whether the task needs to be accounted as such. ->in_iowait_acct nests inside of ->in_iowait, both for efficiency reasons, but also so that the relationship between the two is clear. A waiter may set ->in_wait alone and not care about the accounting. Existing users of nr_iowait() for accounting purposes are switched to use nr_iowait_acct(), which leaves the governor using nr_iowait() as it only cares about iowaiters, not the accounting side. Utilize that there's enough space in rq->nr_iowait to store both values in there, shifting the accounting side by half the size of the type. Thank you to Thomas Gleixner for that [1] suggestion. [1] https://lore.kernel.org/lkml/87sf1b6o9w.ffs@tglx/ Signed-off-by: Jens Axboe --- arch/s390/appldata/appldata_base.c | 2 +- arch/s390/appldata/appldata_os.c | 2 +- fs/proc/stat.c | 2 +- include/linux/sched.h | 6 +++ include/linux/sched/stat.h | 5 ++- kernel/sched/core.c | 62 +++++++++++++++++++++++------- kernel/sched/sched.h | 1 + kernel/time/tick-sched.c | 6 +-- 8 files changed, 65 insertions(+), 21 deletions(-) diff --git a/arch/s390/appldata/appldata_base.c b/arch/s390/appldata/applda= ta_base.c index c2978cb03b36..6844b5294a8b 100644 --- a/arch/s390/appldata/appldata_base.c +++ b/arch/s390/appldata/appldata_base.c @@ -423,4 +423,4 @@ EXPORT_SYMBOL_GPL(si_swapinfo); #endif EXPORT_SYMBOL_GPL(nr_threads); EXPORT_SYMBOL_GPL(nr_running); -EXPORT_SYMBOL_GPL(nr_iowait); +EXPORT_SYMBOL_GPL(nr_iowait_acct); diff --git a/arch/s390/appldata/appldata_os.c b/arch/s390/appldata/appldata= _os.c index a363d30ce739..fa4b278aca6c 100644 --- a/arch/s390/appldata/appldata_os.c +++ b/arch/s390/appldata/appldata_os.c @@ -100,7 +100,7 @@ static void appldata_get_os_data(void *data) =20 os_data->nr_threads =3D nr_threads; os_data->nr_running =3D nr_running(); - os_data->nr_iowait =3D nr_iowait(); + os_data->nr_iowait =3D nr_iowait_acct(); os_data->avenrun[0] =3D avenrun[0] + (FIXED_1/200); os_data->avenrun[1] =3D avenrun[1] + (FIXED_1/200); os_data->avenrun[2] =3D avenrun[2] + (FIXED_1/200); diff --git a/fs/proc/stat.c b/fs/proc/stat.c index da60956b2915..149be7a884fb 100644 --- a/fs/proc/stat.c +++ b/fs/proc/stat.c @@ -180,7 +180,7 @@ static int show_stat(struct seq_file *p, void *v) (unsigned long long)boottime.tv_sec, total_forks, nr_running(), - nr_iowait()); + nr_iowait_acct()); =20 seq_put_decimal_ull(p, "softirq ", (unsigned long long)sum_softirq); =20 diff --git a/include/linux/sched.h b/include/linux/sched.h index dcfc2830ed8e..26c69db5484b 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -924,7 +924,13 @@ struct task_struct { =20 /* Bit to tell TOMOYO we're in execve(): */ unsigned in_execve:1; + /* task is in iowait */ unsigned in_iowait:1; + /* + * task is in iowait and should be accounted as such. can only be set + * if ->in_iowait is also set. + */ + unsigned in_iowait_acct:1; #ifndef TIF_RESTORE_SIGMASK unsigned restore_sigmask:1; #endif diff --git a/include/linux/sched/stat.h b/include/linux/sched/stat.h index 0108a38bb64d..31e8a44b3d71 100644 --- a/include/linux/sched/stat.h +++ b/include/linux/sched/stat.h @@ -19,8 +19,9 @@ DECLARE_PER_CPU(unsigned long, process_counts); extern int nr_processes(void); extern unsigned int nr_running(void); extern bool single_task_running(void); -extern unsigned int nr_iowait(void); -extern unsigned int nr_iowait_cpu(int cpu); +unsigned int nr_iowait_acct(void); +unsigned int nr_iowait_acct_cpu(int cpu); +unsigned int nr_iowait_cpu(int cpu); =20 static inline int sched_info_on(void) { diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 63f6d44f460c..d52d3118dd73 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3721,30 +3721,47 @@ static inline cpumask_t *alloc_user_cpus_ptr(int no= de) =20 #endif /* !CONFIG_SMP */ =20 +/* + * Store iowait and iowait_acct state in the same variable. The lower bits + * hold the iowait state, and the upper bits hold the iowait_acct state. + */ static void task_iowait_inc(struct task_struct *p) { #ifdef CONFIG_64BIT - atomic_long_inc(&task_rq(p)->nr_iowait); + long val =3D 1 + ((long) p->in_iowait_acct << 32); + atomic_long_add(val, &task_rq(p)->nr_iowait); #else - atomic_inc(&task_rq(p)->nr_iowait); + int val =3D 1 + ((int) p->in_iowait_acct << 16); + atomic_add(val, &task_rq(p)->nr_iowait); #endif } =20 static void task_iowait_dec(struct task_struct *p) { #ifdef CONFIG_64BIT - atomic_long_dec(&task_rq(p)->nr_iowait); + long val =3D 1 + ((long) p->in_iowait_acct << 32); + atomic_long_sub(val, &task_rq(p)->nr_iowait); #else - atomic_dec(&task_rq(p)->nr_iowait); + int val =3D 1 + ((int) p->in_iowait_acct << 16); + atomic_sub(val, &task_rq(p)->nr_iowait); #endif } =20 int rq_iowait(struct rq *rq) { #ifdef CONFIG_64BIT - return atomic_long_read(&rq->nr_iowait); + return atomic_long_read(&rq->nr_iowait) & ((1UL << 32) - 1); +#else + return atomic_read(&rq->nr_iowait) & ((1U << 16) - 1); +#endif +} + +int rq_iowait_acct(struct rq *rq) +{ +#ifdef CONFIG_64BIT + return atomic_long_read(&rq->nr_iowait) >> 32; #else - return atomic_read(&rq->nr_iowait); + return atomic_read(&rq->nr_iowait) >> 16; #endif } =20 @@ -5497,7 +5514,12 @@ unsigned long long nr_context_switches(void) * it does become runnable. */ =20 -unsigned int nr_iowait_cpu(int cpu) +unsigned int nr_iowait_acct_cpu(int cpu) +{ + return rq_iowait_acct(cpu_rq(cpu)); +} + +unsigned nr_iowait_cpu(int cpu) { return rq_iowait(cpu_rq(cpu)); } @@ -5532,12 +5554,12 @@ unsigned int nr_iowait_cpu(int cpu) * Task CPU affinities can make all that even more 'interesting'. */ =20 -unsigned int nr_iowait(void) +unsigned int nr_iowait_acct(void) { unsigned int i, sum =3D 0; =20 for_each_possible_cpu(i) - sum +=3D nr_iowait_cpu(i); + sum +=3D nr_iowait_acct_cpu(i); =20 return sum; } @@ -9032,18 +9054,32 @@ int __sched yield_to(struct task_struct *p, bool pr= eempt) } EXPORT_SYMBOL_GPL(yield_to); =20 +/* + * Returns a token which is comprised of the two bits of iowait wait state= - + * one is whether we're making ourselves as in iowait for cpufreq reasons, + * and the other is if the task should be accounted as such. + */ long io_schedule_prepare(void) { - long old_iowait =3D current->in_iowait; - +#ifdef CONFIG_64BIT + long token =3D current->in_iowait + ((long) current->in_iowait_acct << 32= ); +#else + int token =3D current->in_iowait + (current->in_iowait_acct << 16); +#endif current->in_iowait =3D 1; + current->in_iowait_acct =3D 1; blk_flush_plug(current->plug, true); - return old_iowait; + return token; } =20 void io_schedule_finish(long token) { - current->in_iowait =3D token; + current->in_iowait =3D token & 0x01; +#ifdef CONFIG_64BIT + current->in_iowait_acct =3D token >> 32; +#else + current->in_iowait_acct =3D token >> 16; +#endif } =20 /* diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index c2802d066615..85fdd6028682 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -3478,5 +3478,6 @@ extern u64 avg_vruntime(struct cfs_rq *cfs_rq); extern int entity_eligible(struct cfs_rq *cfs_rq, struct sched_entity *se); =20 int rq_iowait(struct rq *rq); +int rq_iowait_acct(struct rq *rq); =20 #endif /* _KERNEL_SCHED_SCHED_H */ diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 269e21590df5..52f377c5871d 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -728,7 +728,7 @@ static void tick_nohz_stop_idle(struct tick_sched *ts, = ktime_t now) delta =3D ktime_sub(now, ts->idle_entrytime); =20 write_seqcount_begin(&ts->idle_sleeptime_seq); - if (nr_iowait_cpu(smp_processor_id()) > 0) + if (nr_iowait_acct_cpu(smp_processor_id()) > 0) ts->iowait_sleeptime =3D ktime_add(ts->iowait_sleeptime, delta); else ts->idle_sleeptime =3D ktime_add(ts->idle_sleeptime, delta); @@ -801,7 +801,7 @@ u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time) struct tick_sched *ts =3D &per_cpu(tick_cpu_sched, cpu); =20 return get_cpu_sleep_time_us(ts, &ts->idle_sleeptime, - !nr_iowait_cpu(cpu), last_update_time); + !nr_iowait_acct_cpu(cpu), last_update_time); } EXPORT_SYMBOL_GPL(get_cpu_idle_time_us); =20 @@ -827,7 +827,7 @@ u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_ti= me) struct tick_sched *ts =3D &per_cpu(tick_cpu_sched, cpu); =20 return get_cpu_sleep_time_us(ts, &ts->iowait_sleeptime, - nr_iowait_cpu(cpu), last_update_time); + nr_iowait_acct_cpu(cpu), last_update_time); } EXPORT_SYMBOL_GPL(get_cpu_iowait_time_us); =20 --=20 2.43.0