From nobody Tue Dec 16 18:22:51 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CEEFBC77B76 for ; Fri, 21 Apr 2023 17:40:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233235AbjDURkm (ORCPT ); Fri, 21 Apr 2023 13:40:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40450 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233178AbjDURk0 (ORCPT ); Fri, 21 Apr 2023 13:40:26 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2C01112C96 for ; Fri, 21 Apr 2023 10:40:25 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id 98e67ed59e1d1-247c0ee87aaso1395325a91.0 for ; Fri, 21 Apr 2023 10:40:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682098824; x=1684690824; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=abvwWBWOn0vLHIokwPNC/tHyu1Fa1R/g8ZqTWIaeK3o=; b=XiTBjh0uy9kBhESmjZrsmo27Fd2heaOUEeGy+wI7JgY+55BFi83KpJfUElFkBAdtm2 /9Y+dOWIDQQOwVTZr/9l2zpPw4NhLuSad5LwurXNF9rM5PxMETajnAk6IdSR9itjleO5 5RCZOU6uWoQn/XlBfG6L6Ut73QqfPkSzpxRKMZq9c5dNC03EQysfXGAmQsX7vQIaMlXI wrYrPEcgEz4/6R89vEXhItTLbmEid9U911aGVkkoSTQdm4KxH3ssyAravljY6M6dDyLy P40RfGznmXP1ykBLl7M1nQBK+thu7rJX9Fwoe+MatBRuu3PDwmPEoG+B+msi8iGqptyL Fwng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682098824; x=1684690824; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=abvwWBWOn0vLHIokwPNC/tHyu1Fa1R/g8ZqTWIaeK3o=; b=HE8BaaYa8bU9hIZTAP50ONOs5zzrOWwjw6gR0sx4aDhAtDZopqVVqU0Fsi6UzTgPKc xejn7iDZZsqIseDy/+re7ocegd3sL9wvO9DXB6WQTcuYwm09fduOU3PKAj0vEIp9NG5z VRGwZZSzS+WzxOCdwqpB2XJtaLWDGC9Ld2Qw8Os1aROYf1XAALYhlclqeNrU8GG2zAId uLIA02C5475/oYLxRCgOaoAOQqXlTrzJvv9Ve8m3OS1taOuMnhBnGucbzqQhtz5MFe4S 7zPy90QuGpyc/37w3QRFUMBE4cJxDCfmP3CjZ03+vpgHxGKLQvStvD+br12IYileotGB hIoQ== X-Gm-Message-State: AAQBX9dBK/WLxdtQpM0RWnHgxjz19dzyCAVZC1Wm2kev0raWE+t0Y5x0 K+gzzChQMTEDJGgtKuzWnrft7wxq1l64TlkM X-Google-Smtp-Source: AKy350YbjANQ50zXR9lpXjPZgwQ40mwqLJ7O/YqrPonNpx/xppRCva3J0ueIG/FDAbEMT7LPHpueSbGCVpFSyjJ8 X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:90a:290:b0:247:2677:77b with SMTP id w16-20020a17090a029000b002472677077bmr1455398pja.8.1682098824669; Fri, 21 Apr 2023 10:40:24 -0700 (PDT) Date: Fri, 21 Apr 2023 17:40:16 +0000 In-Reply-To: <20230421174020.2994750-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230421174020.2994750-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.634.g4ca3ef3211-goog Message-ID: <20230421174020.2994750-2-yosryahmed@google.com> Subject: [PATCH v5 1/5] writeback: move wb_over_bg_thresh() call outside lock section From: Yosry Ahmed To: Alexander Viro , Christian Brauner , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Tejun Heo Cc: Jan Kara , Jens Axboe , "=?UTF-8?q?Michal=20Koutn=C3=BD?=" , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Yosry Ahmed Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" wb_over_bg_thresh() calls mem_cgroup_wb_stats() which invokes an rstat flush, which can be expensive on large systems. Currently, wb_writeback() calls wb_over_bg_thresh() within a lock section, so we have to do the rstat flush atomically. On systems with a lot of cpus and/or cgroups, this can cause us to disable irqs for a long time, potentially causing problems. Move the call to wb_over_bg_thresh() outside the lock section in preparation to make the rstat flush in mem_cgroup_wb_stats() non-atomic. The list_empty(&wb->work_list) check should be okay outside the lock section of wb->list_lock as it is protected by a separate lock (wb->work_lock), and wb_over_bg_thresh() doesn't seem like it is modifying any of wb->b_* lists the wb->list_lock is protecting. Also, the loop seems to be already releasing and reacquring the lock, so this refactoring looks safe. Signed-off-by: Yosry Ahmed Reviewed-by: Michal Koutn=C3=BD Reviewed-by: Jan Kara Acked-by: Shakeel Butt Acked-by: Tejun Heo --- fs/fs-writeback.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c index 1db3e3c24b43..11aa1652fb84 100644 --- a/fs/fs-writeback.c +++ b/fs/fs-writeback.c @@ -2024,7 +2024,6 @@ static long wb_writeback(struct bdi_writeback *wb, struct blk_plug plug; =20 blk_start_plug(&plug); - spin_lock(&wb->list_lock); for (;;) { /* * Stop writeback when nr_pages has been consumed @@ -2049,6 +2048,9 @@ static long wb_writeback(struct bdi_writeback *wb, if (work->for_background && !wb_over_bg_thresh(wb)) break; =20 + + spin_lock(&wb->list_lock); + /* * Kupdate and background works are special and we want to * include all inodes that need writing. Livelock avoidance is @@ -2078,13 +2080,19 @@ static long wb_writeback(struct bdi_writeback *wb, * mean the overall work is done. So we keep looping as long * as made some progress on cleaning pages or inodes. */ - if (progress) + if (progress) { + spin_unlock(&wb->list_lock); continue; + } + /* * No more inodes for IO, bail */ - if (list_empty(&wb->b_more_io)) + if (list_empty(&wb->b_more_io)) { + spin_unlock(&wb->list_lock); break; + } + /* * Nothing written. Wait for some inode to * become available for writeback. Otherwise @@ -2096,9 +2104,7 @@ static long wb_writeback(struct bdi_writeback *wb, spin_unlock(&wb->list_lock); /* This function drops i_lock... */ inode_sleep_on_writeback(inode); - spin_lock(&wb->list_lock); } - spin_unlock(&wb->list_lock); blk_finish_plug(&plug); =20 return nr_pages - work->nr_pages; --=20 2.40.0.634.g4ca3ef3211-goog From nobody Tue Dec 16 18:22:51 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6574C77B78 for ; Fri, 21 Apr 2023 17:40:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233372AbjDURkq (ORCPT ); Fri, 21 Apr 2023 13:40:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40476 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232300AbjDURk2 (ORCPT ); Fri, 21 Apr 2023 13:40:28 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EE9E612582 for ; Fri, 21 Apr 2023 10:40:26 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id 98e67ed59e1d1-24736ac595aso2200461a91.1 for ; Fri, 21 Apr 2023 10:40:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682098826; x=1684690826; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=ze/kPlw8gotaEo1sJnB2fYrZx8MA/zlu9x0xYHZ+g7E=; b=evkdgDpSyqlLAXON2ZuU2lSGfLM1dQnhdPP+fmyZWeKwGNZTWFCJKPVGhM2bZZGKOC f7YfSGCo2Nf3oXwwYJmGjEDgBvFsw6i76b70QgcRaecrvTKD295qL9CqwnO4UztiMSAR wca4o1fBiZOb2DZIrqp+VDQfAwIcLci5K41MPeROy0/47HRyQgXuMC8OBwZyCxCgwh/S 8f4JUGNY1xeou8G6F0i+W3dKhEWQ3Zv+nckYooRRkUzwRcW77nDMDOwXtSb1IOOWcx0Y jIqwdtbmW/rhz4VKzZZ8k/jeg5TikDCeSv0Wmg2wr9UElKoOoUpGS9AakxlgHuAC0y31 KsJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682098826; x=1684690826; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=ze/kPlw8gotaEo1sJnB2fYrZx8MA/zlu9x0xYHZ+g7E=; b=EWXdK/obcCOZsOmX7JuJxP3nu6RPMu4Vopb6pSXd994RHzHGx0NGjylXq+M/2qrNXb ot0a1hFmJgC6aAkjufdXZ8zvcxzemUfHs7N0oSUPDTWmd7806BgNmQZpuMb12NV2W2iH GOxUA9STuQ/hBNbNIlD8MTQXKB2lql1ZhtdvCfda7s1E4MFBx5BhRJs3Gm+OgI8ix3FV 8hm43kIUFySARAFUrm064iPTWphMD0vrvoNbpFlVQUkQciu68kQ3FiYpOWSFn7DEsdgO arApE1Edzg2rwm+KMYwwE0s8qSMFI4eWZGjosfdmU0av39YA40PKVt5dIfc9z0wjlYF9 v06w== X-Gm-Message-State: AAQBX9d8mwtZrnCGr/haoRja+nQL8SnE9wGF5Nep54y/QLVM/BRuPjkh tfGryJMrKrufwJIHWvwlJt8KkuO0+PBQA8Kh X-Google-Smtp-Source: AKy350aoZj3p9w7xLo2ekZAFCbJRpr+XbYHOfgfJjD/OnpfpL06mwTgu6bL4JQHk39cy27rMag7V9/fdDGHimXJS X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:90a:3fcd:b0:24b:27db:3cfc with SMTP id u13-20020a17090a3fcd00b0024b27db3cfcmr1409596pjm.8.1682098826505; Fri, 21 Apr 2023 10:40:26 -0700 (PDT) Date: Fri, 21 Apr 2023 17:40:17 +0000 In-Reply-To: <20230421174020.2994750-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230421174020.2994750-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.634.g4ca3ef3211-goog Message-ID: <20230421174020.2994750-3-yosryahmed@google.com> Subject: [PATCH v5 2/5] memcg: flush stats non-atomically in mem_cgroup_wb_stats() From: Yosry Ahmed To: Alexander Viro , Christian Brauner , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Tejun Heo Cc: Jan Kara , Jens Axboe , "=?UTF-8?q?Michal=20Koutn=C3=BD?=" , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Yosry Ahmed Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The previous patch moved the wb_over_bg_thresh()->mem_cgroup_wb_stats() code path in wb_writeback() outside the lock section. We no longer need to flush the stats atomically. Flush the stats non-atomically. Signed-off-by: Yosry Ahmed Reviewed-by: Michal Koutn=C3=BD Acked-by: Shakeel Butt Acked-by: Tejun Heo --- mm/memcontrol.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4b27e245a055..5e79fdf8442b 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4648,11 +4648,7 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, u= nsigned long *pfilepages, struct mem_cgroup *memcg =3D mem_cgroup_from_css(wb->memcg_css); struct mem_cgroup *parent; =20 - /* - * wb_writeback() takes a spinlock and calls - * wb_over_bg_thresh()->mem_cgroup_wb_stats(). Do not sleep. - */ - mem_cgroup_flush_stats_atomic(); + mem_cgroup_flush_stats(); =20 *pdirty =3D memcg_page_state(memcg, NR_FILE_DIRTY); *pwriteback =3D memcg_page_state(memcg, NR_WRITEBACK); --=20 2.40.0.634.g4ca3ef3211-goog From nobody Tue Dec 16 18:22:51 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1838EC7618E for ; Fri, 21 Apr 2023 17:40:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233375AbjDURku (ORCPT ); Fri, 21 Apr 2023 13:40:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40498 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233290AbjDURkb (ORCPT ); Fri, 21 Apr 2023 13:40:31 -0400 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C619212C96 for ; Fri, 21 Apr 2023 10:40:28 -0700 (PDT) Received: by mail-pg1-x549.google.com with SMTP id 41be03b00d2f7-520f3f18991so1458485a12.3 for ; Fri, 21 Apr 2023 10:40:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682098828; x=1684690828; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=ZwYNf0IQKL13Uvalja3XtUe1fz98Sj9XyCNuQ2xKGcs=; b=Vz7q7OzmTKVhjKuL1Z10tx4sjzaZzQc1RfkV5JIKDa4QF+qc7PE1nRpKj4JbTW26CL 6HWg0Bk/Qn7IeGOg2v+aAU068B55qLODSnp8tJGuESSICbp+6Gjpfa3HASvT4P/INIIX FNJAyhgiBW8P7szSQRrC+9yXtrHR4LwEf5c+uGkIF6nWGq5t81yHqYblmUSNV6/Pj4a5 x4hPCDxoK7EryyUSzUifkrvbv6JClx5/LSyTe0bPdsXuYCvvh5cuu7sfplfgaUZyWnve cPw/375qmd31nmxG3Xi1PRi7CTxei8qHjgER9ienP6JVZlSSk1rMC1yFsifBD6+eDFuk /A6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682098828; x=1684690828; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=ZwYNf0IQKL13Uvalja3XtUe1fz98Sj9XyCNuQ2xKGcs=; b=gFPejYgdW72FEgimUZj+m6qd51Jia/Oum69pWEP8PCbH4CtsqFJpAPQex6WYPiAYfQ +wiDEJ7i265OUfXwmClDm7vbrHd5jv3Wr/2QqAv0YN+004KwXpjY2IkYp2V+602eQR9d HCRZIE2PCIhAp6z31OUPFqss+bfARyzJkiqGn+4LvFi3rCBDmLa4hUxO3IdKh4krPth7 M11EOGGzCVBh5nZLvxgPtONdghIrtilrc4McgoGiEt6DbKNgO0nLLrIFgG7HyUeIw8Qs VUvJI9U2OSNMtcaJLK4hunJE5IiTRPpkGA2NxQQqM4dlsvWYKkOfgUg0PjhdWguDQEb6 j1Eg== X-Gm-Message-State: AAQBX9cm2JSUKf9HwCK9qre6k7WwM0s1mY/dTrKaKk5xSaDHZaJPNNyi EFacF30X6co8/uBH6GYWviPwKA0iTZBPtoNJ X-Google-Smtp-Source: AKy350aINzx6WzxMD77WpFdXBe8LNCyI2424id24uJ5dWpjeDUwupScbzRciRo/Uc9LOO+muamaNQYT7/vZDgFTY X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:903:2447:b0:1a9:2c3e:b087 with SMTP id l7-20020a170903244700b001a92c3eb087mr1970101pls.0.1682098828387; Fri, 21 Apr 2023 10:40:28 -0700 (PDT) Date: Fri, 21 Apr 2023 17:40:18 +0000 In-Reply-To: <20230421174020.2994750-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230421174020.2994750-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.634.g4ca3ef3211-goog Message-ID: <20230421174020.2994750-4-yosryahmed@google.com> Subject: [PATCH v5 3/5] memcg: calculate root usage from global state From: Yosry Ahmed To: Alexander Viro , Christian Brauner , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Tejun Heo Cc: Jan Kara , Jens Axboe , "=?UTF-8?q?Michal=20Koutn=C3=BD?=" , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Yosry Ahmed Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Currently, we approximate the root usage by adding the memcg stats for anon, file, and conditionally swap (for memsw). To read the memcg stats we need to invoke an rstat flush. rstat flushes can be expensive, they scale with the number of cpus and cgroups on the system. mem_cgroup_usage() is called by memcg_events()->mem_cgroup_threshold() with irqs disabled, so such an expensive operation with irqs disabled can cause problems. Instead, approximate the root usage from global state. This is not 100% accurate, but the root usage has always been ill-defined anyway. Signed-off-by: Yosry Ahmed Reviewed-by: Michal Koutn=C3=BD Acked-by: Shakeel Butt --- mm/memcontrol.c | 24 +++++------------------- 1 file changed, 5 insertions(+), 19 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 5e79fdf8442b..cb78bba5b4a4 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3699,27 +3699,13 @@ static unsigned long mem_cgroup_usage(struct mem_cg= roup *memcg, bool swap) =20 if (mem_cgroup_is_root(memcg)) { /* - * We can reach here from irq context through: - * uncharge_batch() - * |--memcg_check_events() - * |--mem_cgroup_threshold() - * |--__mem_cgroup_threshold() - * |--mem_cgroup_usage - * - * rstat flushing is an expensive operation that should not be - * done from irq context; use stale stats in this case. - * Arguably, usage threshold events are not reliable on the root - * memcg anyway since its usage is ill-defined. - * - * Additionally, other call paths through memcg_check_events() - * disable irqs, so make sure we are flushing stats atomically. + * Approximate root's usage from global state. This isn't + * perfect, but the root usage was always an approximation. */ - if (in_task()) - mem_cgroup_flush_stats_atomic(); - val =3D memcg_page_state(memcg, NR_FILE_PAGES) + - memcg_page_state(memcg, NR_ANON_MAPPED); + val =3D global_node_page_state(NR_FILE_PAGES) + + global_node_page_state(NR_ANON_MAPPED); if (swap) - val +=3D memcg_page_state(memcg, MEMCG_SWAP); + val +=3D total_swap_pages - get_nr_swap_pages(); } else { if (!swap) val =3D page_counter_read(&memcg->memory); --=20 2.40.0.634.g4ca3ef3211-goog From nobody Tue Dec 16 18:22:51 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2496EC7618E for ; Fri, 21 Apr 2023 17:40:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233380AbjDURky (ORCPT ); Fri, 21 Apr 2023 13:40:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40524 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233293AbjDURkc (ORCPT ); Fri, 21 Apr 2023 13:40:32 -0400 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A8B6312588 for ; Fri, 21 Apr 2023 10:40:30 -0700 (PDT) Received: by mail-pf1-x44a.google.com with SMTP id d2e1a72fcca58-63b6bd2eebaso1799541b3a.1 for ; Fri, 21 Apr 2023 10:40:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682098830; x=1684690830; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2xVbU1EXT3F/5uSzRhbvwRdAhlPEIhLbnGkOO/O2kc0=; b=VsD7UvvlQQthDrDrmrxfA+6j9bxs+fBOtiWfjPuVqKRncTMVYwHyMK7lQKw4z7d6OO 8f3IhH6XmGqVDVmFY8Cyv6LrhOYmMlAEwO97OjVKp4EfQ7GvmXRUQBA2elyZi2HdAWT8 tC3JhABpjBI1uwry+YoMsz8jQ+5FjhTBlcxyLfQuh9MawFFcJmVXlkCxxE83KRAGfIRN izUhapvRy65QRBOUgMasmNbiSkr8HJo7bR2eGmnfXoyNgGu2rdwIe+txD2JLoEKv7Tja ItjbWxT/dXoPWnPJUuXxJqFt3IyOH6lsotFnhAXC0TTQySKSNCNtSjYyYHqymn6WRMKQ FgWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682098830; x=1684690830; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2xVbU1EXT3F/5uSzRhbvwRdAhlPEIhLbnGkOO/O2kc0=; b=c+Wkc8tN3ZggZBxjCjd2SDX0WvLvZlsz/epZBGBlsijzeNBj+n9V4kvnIsRxxxVCFF /wxU+KxhazM4g8SKoPm3/kA9bnBA/zhcNvwM0ua96oDg400nTWgxKWBSZJISmtZ8cY46 bvamz0FoTi0LHehvVGI5FnZAzX8A9dpTXMd8he+CQMUUwvZ8YzUYgq3dXppEts2exv+s TgvtvJNSutZY44ZCuTgdMCUzPB7vC3H8iTRoYW83f0EomN8zKV+Dk0J1gwhlm9XTDobY Y5UlEpXsHrSe0hphQMh8fCZOo5ShZYuHG5dLZC1Bf345TMEIGCc1vP1nE0Fa5Vboo8Oe e5hw== X-Gm-Message-State: AAQBX9cedyP3EHPc7imBfgS1I98/7c/sqdePW4/mWeQVv4+Q27cCkqZu x35mhkEZIjEYKkXUc1mWn2mSiRod1w1/hf8Z X-Google-Smtp-Source: AKy350a5A112WsBcfM4E2oTlYZXb/Q86xb8XlqVuh7OqAqi/kCn371in9+BxtMemeJoGBiiS6Eu5MnjLStpGGXAY X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6a00:2e97:b0:62d:9bea:2a0c with SMTP id fd23-20020a056a002e9700b0062d9bea2a0cmr2546450pfb.4.1682098830169; Fri, 21 Apr 2023 10:40:30 -0700 (PDT) Date: Fri, 21 Apr 2023 17:40:19 +0000 In-Reply-To: <20230421174020.2994750-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230421174020.2994750-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.634.g4ca3ef3211-goog Message-ID: <20230421174020.2994750-5-yosryahmed@google.com> Subject: [PATCH v5 4/5] memcg: remove mem_cgroup_flush_stats_atomic() From: Yosry Ahmed To: Alexander Viro , Christian Brauner , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Tejun Heo Cc: Jan Kara , Jens Axboe , "=?UTF-8?q?Michal=20Koutn=C3=BD?=" , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Yosry Ahmed Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Previous patches removed all callers of mem_cgroup_flush_stats_atomic(). Remove the function and simplify the code. Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt --- include/linux/memcontrol.h | 5 ----- mm/memcontrol.c | 24 +++++------------------- 2 files changed, 5 insertions(+), 24 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 222d7370134c..00a88cf947e1 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1038,7 +1038,6 @@ static inline unsigned long lruvec_page_state_local(s= truct lruvec *lruvec, } =20 void mem_cgroup_flush_stats(void); -void mem_cgroup_flush_stats_atomic(void); void mem_cgroup_flush_stats_ratelimited(void); =20 void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item i= dx, @@ -1537,10 +1536,6 @@ static inline void mem_cgroup_flush_stats(void) { } =20 -static inline void mem_cgroup_flush_stats_atomic(void) -{ -} - static inline void mem_cgroup_flush_stats_ratelimited(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index cb78bba5b4a4..eadfe371a8f2 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -639,7 +639,7 @@ static inline void memcg_rstat_updated(struct mem_cgrou= p *memcg, int val) } } =20 -static void do_flush_stats(bool atomic) +static void do_flush_stats(void) { /* * We always flush the entire tree, so concurrent flushers can just @@ -652,30 +652,16 @@ static void do_flush_stats(bool atomic) =20 WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME); =20 - if (atomic) - cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup); - else - cgroup_rstat_flush(root_mem_cgroup->css.cgroup); + cgroup_rstat_flush(root_mem_cgroup->css.cgroup); =20 atomic_set(&stats_flush_threshold, 0); atomic_set(&stats_flush_ongoing, 0); } =20 -static bool should_flush_stats(void) -{ - return atomic_read(&stats_flush_threshold) > num_online_cpus(); -} - void mem_cgroup_flush_stats(void) { - if (should_flush_stats()) - do_flush_stats(false); -} - -void mem_cgroup_flush_stats_atomic(void) -{ - if (should_flush_stats()) - do_flush_stats(true); + if (atomic_read(&stats_flush_threshold) > num_online_cpus()) + do_flush_stats(); } =20 void mem_cgroup_flush_stats_ratelimited(void) @@ -690,7 +676,7 @@ static void flush_memcg_stats_dwork(struct work_struct = *w) * Always flush here so that flushing in latency-sensitive paths is * as cheap as possible. */ - do_flush_stats(false); + do_flush_stats(); queue_delayed_work(system_unbound_wq, &stats_flush_dwork, FLUSH_TIME); } =20 --=20 2.40.0.634.g4ca3ef3211-goog From nobody Tue Dec 16 18:22:51 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A280C77B76 for ; Fri, 21 Apr 2023 17:40:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232791AbjDURk5 (ORCPT ); Fri, 21 Apr 2023 13:40:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40552 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233318AbjDURke (ORCPT ); Fri, 21 Apr 2023 13:40:34 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B9CD512C96 for ; Fri, 21 Apr 2023 10:40:32 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-b922a14e089so3322273276.2 for ; Fri, 21 Apr 2023 10:40:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682098832; x=1684690832; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xEOFJLcgICBTmNTLnyQD1ooBVf7w9cMBHK4zBwAxhps=; b=2LLI0uS0HCGgUG+To5FG73zbxk+tuzk5tTcOBZNGKwnVbYk9yq2r0zkyTl0MD4wfQ3 kKiggVfa7NNclULg8NuxAqHV3zYnq7LZjfMrcVD62yChQCNHWbk54zVsBhxomTb0t0SN +XgJr7wul3fQD2JopF/MZfktxPtuUWKER8IL9ZU+RSEl7sW66jFXxgj6K5ErZztO7Nce l4ZWAyWNanyDDUgKfIgOcWOkpHNwJKJvYyr6/84gH8i9GqoBVPKeVaSusHPk+GdeSMBt A9hidHBqS6YCIVpadKtHiQ8uYxCY8mC3l7iI3bjfr9izY/KQoToKFIDt3lP79eIF321D fMOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682098832; x=1684690832; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xEOFJLcgICBTmNTLnyQD1ooBVf7w9cMBHK4zBwAxhps=; b=HzRxe58rlppsG17pmttBxLwlASnYQQ4eEbDVOPjLvLy+3xSc5RDvNUspzG7x/UjPQ1 zNs1VsxEcCXCf1ISFzJhKdkvT4+fjsBQxGIYEF1NHf+MZ32ZObQPQvi1mBGHadP/xgkS fHz+SIA3pTBu7lpFxiO+PfHh3BqCCo6JFKjYyGDj4OkH5yr6nhpf6GI+WrrMmLDJFz0D w07Fx5XuqjE0lR/Ioqv8/M7y4h94tQ4y+2i+nnVEB5aObnOURcLnhDC+pjZMbGB9R4+R jukJphhxDF6W6I9fxZdyYiq7nolXZWbTpyuWvRFTFNyyE2COgWybkoOuAqXr7lrQ4Pno 9C/A== X-Gm-Message-State: AAQBX9fnlpdw2MYEFHdE6JSpDibH9ONKPUFUmsmnD+WYFyH4XedzRgoo cgAPtRQt+zFsnXuuo0CJNqm1lHdcKJza5oJ8 X-Google-Smtp-Source: AKy350ZZ8ts19Uuwp1KvgVZn+d+WSE2zsbcOTm9kgrCL3PJuuy6na6YJxn3jgGZppB+bAg1Vc1I9v8GZpcBWJ5vX X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a25:bc4a:0:b0:b33:531b:3dd4 with SMTP id d10-20020a25bc4a000000b00b33531b3dd4mr1583021ybk.1.1682098831943; Fri, 21 Apr 2023 10:40:31 -0700 (PDT) Date: Fri, 21 Apr 2023 17:40:20 +0000 In-Reply-To: <20230421174020.2994750-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230421174020.2994750-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.634.g4ca3ef3211-goog Message-ID: <20230421174020.2994750-6-yosryahmed@google.com> Subject: [PATCH v5 5/5] cgroup: remove cgroup_rstat_flush_atomic() From: Yosry Ahmed To: Alexander Viro , Christian Brauner , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Tejun Heo Cc: Jan Kara , Jens Axboe , "=?UTF-8?q?Michal=20Koutn=C3=BD?=" , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Yosry Ahmed Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Previous patches removed the only caller of cgroup_rstat_flush_atomic(). Remove the function and simplify the code. Signed-off-by: Yosry Ahmed --- include/linux/cgroup.h | 1 - kernel/cgroup/rstat.c | 26 +++++--------------------- 2 files changed, 5 insertions(+), 22 deletions(-) diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h index 885f5395fcd0..567c547cf371 100644 --- a/include/linux/cgroup.h +++ b/include/linux/cgroup.h @@ -692,7 +692,6 @@ static inline void cgroup_path_from_kernfs_id(u64 id, c= har *buf, size_t buflen) */ void cgroup_rstat_updated(struct cgroup *cgrp, int cpu); void cgroup_rstat_flush(struct cgroup *cgrp); -void cgroup_rstat_flush_atomic(struct cgroup *cgrp); void cgroup_rstat_flush_hold(struct cgroup *cgrp); void cgroup_rstat_flush_release(void); =20 diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index d3252b0416b6..f9ad33f117c8 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -171,7 +171,7 @@ __weak noinline void bpf_rstat_flush(struct cgroup *cgr= p, __diag_pop(); =20 /* see cgroup_rstat_flush() */ -static void cgroup_rstat_flush_locked(struct cgroup *cgrp, bool may_sleep) +static void cgroup_rstat_flush_locked(struct cgroup *cgrp) __releases(&cgroup_rstat_lock) __acquires(&cgroup_rstat_lock) { int cpu; @@ -207,9 +207,8 @@ static void cgroup_rstat_flush_locked(struct cgroup *cg= rp, bool may_sleep) } raw_spin_unlock_irqrestore(cpu_lock, flags); =20 - /* if @may_sleep, play nice and yield if necessary */ - if (may_sleep && (need_resched() || - spin_needbreak(&cgroup_rstat_lock))) { + /* play nice and yield if necessary */ + if (need_resched() || spin_needbreak(&cgroup_rstat_lock)) { spin_unlock_irq(&cgroup_rstat_lock); if (!cond_resched()) cpu_relax(); @@ -236,25 +235,10 @@ __bpf_kfunc void cgroup_rstat_flush(struct cgroup *cg= rp) might_sleep(); =20 spin_lock_irq(&cgroup_rstat_lock); - cgroup_rstat_flush_locked(cgrp, true); + cgroup_rstat_flush_locked(cgrp); spin_unlock_irq(&cgroup_rstat_lock); } =20 -/** - * cgroup_rstat_flush_atomic- atomic version of cgroup_rstat_flush() - * @cgrp: target cgroup - * - * This function can be called from any context. - */ -void cgroup_rstat_flush_atomic(struct cgroup *cgrp) -{ - unsigned long flags; - - spin_lock_irqsave(&cgroup_rstat_lock, flags); - cgroup_rstat_flush_locked(cgrp, false); - spin_unlock_irqrestore(&cgroup_rstat_lock, flags); -} - /** * cgroup_rstat_flush_hold - flush stats in @cgrp's subtree and hold * @cgrp: target cgroup @@ -269,7 +253,7 @@ void cgroup_rstat_flush_hold(struct cgroup *cgrp) { might_sleep(); spin_lock_irq(&cgroup_rstat_lock); - cgroup_rstat_flush_locked(cgrp, true); + cgroup_rstat_flush_locked(cgrp); } =20 /** --=20 2.40.0.634.g4ca3ef3211-goog