From nobody Fri Feb 13 01:40:26 2026 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B3D4713D2BA for ; Tue, 4 Jun 2024 02:06:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717466772; cv=none; b=YWo3mQZaT5aERDXdejpMnXbGfiZAUFrKPHBTsYDjQFKygQDU+cF/HE4zT2sAogZ2gvD+fdmOPSPG7WO1vHEvxcr+uQcnPXCqsJ0NHbAmSzgcehkk6dMmxEmqFVeLvNBEqQFxVxPKIK97I9kVEfoMvQbNh0XS9oDFyffi5/q4t2k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717466772; c=relaxed/simple; bh=msjeUsgEzzOjOpUWHfas3LCKMX1t6e6yDLl0uelE9mg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=dM3741wbQ/d20iLyB3fGeYj0NL3lncmOd/nOznuiB4tUGyWMAe2oXI//MvWCDy40UcKjg7ahsZeq5sNTTlxRnFWvc7hMsrjfIBM8/iS7A3bm3kGGytruW8KlbXP9eCl/yp+3IMDwv/nx1rzvkV918l8QTR3RklG7tNvHTcnBSdo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--yuanchu.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=taLOJr1h; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--yuanchu.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="taLOJr1h" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dfa744fe2f9so870793276.0 for ; Mon, 03 Jun 2024 19:06:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717466769; x=1718071569; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TjWUm/jwk/hpgAY6bYVVdFyPbNpQM4PAlbOY34rPVUc=; b=taLOJr1helQgcs2JhlL6kdfSnGBKeG/3LVHEiZ9Vdl5nRWjTl5YAUPzrEnyV9eehYF Lxvqu8GQguXVjnt+E+h3sKyGTdazv6oYbh8jbATK48mqlft9bxuGvXpzkMlcyxx0SnvY uK3Fo+k11KYaD5fnNdh0XSdJXpm0XxUECkpBf5a9g/8WjBtjJH7prgII0v8mCsdMx+sB 6HOcbPdetq7ONLQaH6i/lEDQVMtbHmv66iCpffsbl3f64k351lwEFGmYo0KLSjcYCVLu Y26QnOD9lh4PL5F7kOCHx3MAeDrCH9lAaXONfMbsCVOuGbF2hygVxJUi0b9fw26gH59t xUdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717466769; x=1718071569; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TjWUm/jwk/hpgAY6bYVVdFyPbNpQM4PAlbOY34rPVUc=; b=R8qBfC2H1w7RyE2ybp0wQ965IknmrUwW5nM+DX6C/f7CypeRFablcSp0vVx4dccBMt xWZYKBLTe0THNjPcjuAdHMGyJxEysj7ktbqmbOA4VYGVZgDNxm6Z7upL6dqEkvg3WNao 3jNpvvXrO3aIkzTRhnrZUtvSBF+nh/ndW3eNAaDGspr1zUxk/akak4vEjUS/kEKve75c O76Hw9PQ5qmt3HMTc2e+C5ZJUBL90R9rM+PR4jgAlHs/9VKMqFxlNKUfYj/EQAgHDZpj QWZ91W6TlhSHXxVv5pPjCrnV69M3ZcwBJi+dImFrbz34kpl8i+AAabgQLL40ONZADGUA zEwQ== X-Forwarded-Encrypted: i=1; AJvYcCV4mL4LurP3ECNNTYKCN1Qq8uPRaD/lszopaGUrAYiTeGRRjfAx0Y0VVFhHo49MamuQ1QMmBEleZ4sgmlcn/RfxPzTmnCOBqUM28hhk X-Gm-Message-State: AOJu0YynT3ZS2sPvLfWTOIonQ142AAQRqzfvPZN0REhmL6OqRwMqLGx0 Cb1Sw7bcSr0+DQ5t9njrL+UHCdVk8NtZWVoLNguDOub7547tEYXXN/8GtE/dkcYKwbxZueyXZvT GxZ/2xw== X-Google-Smtp-Source: AGHT+IEDfb+3JEeKpiUuy5LGS6SnXp9v414pUptf89nU5rjiLVpz5pdRpN0kM4+F2bCDbN9n10wQa2F2tJMo X-Received: from yuanchu-desktop.svl.corp.google.com ([2620:15c:2a3:200:367f:7387:3dd2:73f1]) (user=yuanchu job=sendgmr) by 2002:a05:6902:110d:b0:df1:d00c:130c with SMTP id 3f1490d57ef6-dfa73bf59bcmr1176974276.5.1717466768619; Mon, 03 Jun 2024 19:06:08 -0700 (PDT) Date: Mon, 3 Jun 2024 19:05:42 -0700 In-Reply-To: <20240604020549.1017540-1-yuanchu@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240604020549.1017540-1-yuanchu@google.com> X-Mailer: git-send-email 2.45.1.467.gbab1589fc0-goog Message-ID: <20240604020549.1017540-2-yuanchu@google.com> Subject: [PATCH v2 1/8] mm: multi-gen LRU: ignore non-leaf pmd_young for force_scan=true From: Yuanchu Xie To: David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Muhammad Usama Anjum Cc: Kalesh Singh , Wei Xu , David Rientjes , Greg Kroah-Hartman , "Rafael J. Wysocki" , Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Shuah Khan , Yosry Ahmed , Matthew Wilcox , Sudarshan Rajagopalan , Kairui Song , "Michael S. Tsirkin" , Vasily Averin , Nhat Pham , Miaohe Lin , Qi Zheng , Abel Wu , "Vishal Moola (Oracle)" , Kefeng Wang , Yuanchu Xie , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When non-leaf pmd accessed bits are available, MGLRU page table walks can clear the non-leaf pmd accessed bit and ignore the accessed bit on the pte if it's on a different node, skipping a generation update as well. If another scan occurrs on the same node as said skipped pte. the non-leaf pmd accessed bit might remain cleared and the pte accessed bits won't be checked. While this is sufficient for reclaim-driven aging, where the goal is to select a reasonably cold page, the access can be missed when aging proactively for workingset estimation of a of a node/memcg. In more detail, get_pfn_folio returns NULL if the folio's nid !=3D node under scanning, so the page table walk skips processing of said pte. Now the pmd_young flag on this pmd is cleared, and if none of the pte's are accessed before another scan occurrs on the folio's node, the pmd_young check fails and the pte accessed bit is skipped. Since force_scan disables various other optimizations, we check force_scan to ignore the non-leaf pmd accessed bit. Signed-off-by: Yuanchu Xie --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index d55e8d07ffc4..73f3718b33f7 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3548,7 +3548,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long = start, unsigned long end, =20 walk->mm_stats[MM_NONLEAF_TOTAL]++; =20 - if (should_clear_pmd_young()) { + if (!walk->force_scan && should_clear_pmd_young()) { if (!pmd_young(val)) continue; =20 --=20 2.45.1.467.gbab1589fc0-goog From nobody Fri Feb 13 01:40:26 2026 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A5D2A13D635 for ; Tue, 4 Jun 2024 02:06:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717466773; cv=none; b=hYFGCd3bC8k0YY4rtKw4aU441ktVRialNarcaeBOblJULBo/QGnT4n9SVkp0Bek6bPb+n4P+KC4Pj7wlniYiEt3UfXFxVZyG2pzWWVid5lEHDi5NVNkW52RttSUesudFEAaGcnQ5aLn8i96wkFIXcg5teb61eIaqbtCxsBXdAVM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717466773; c=relaxed/simple; bh=6gjefxgwl+BmiMXcPNlzSWG8OicK5aV5LZbN3nsiLYY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=uho6AsfLoPmcf7SfYukVaYZeqhbatjv1EBXLDPNSy6XCYZr/2yDShhq62rTxkO6lpSR4lgvtrAKOYitkUy1VZ3c4sQjarThbFqgOjySANSYaOgiQaRFu+Ma29qJtnJzcjI/LApdMMFdFHy3fAk8ZiWz01wuoy6C5IKEr+/FlTFQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--yuanchu.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=XN2wO07F; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--yuanchu.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="XN2wO07F" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-62777fe7b86so57831287b3.1 for ; Mon, 03 Jun 2024 19:06:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717466771; x=1718071571; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=l2jOiTF4T3isqemcmHaO7N7kxplnAs9PyjvY121bHmg=; b=XN2wO07FfNDK1Ri5LAy3yxk8/rZoccVGinww3CanaoJD4BaRKH7JPYHLAMyCG2Yy/m Aa72ysiWaiP8hjE5aPIGH3ujQIvDc8RdhFHi8CvcGLWiJFyAtqGfFOSGITqrOv2Zt/6/ qNMDZMoFhqcO+ZTHfRRO/V6T9u+7DfQcQaIvPNxHkUbb4gne7eOS76FTJi8WcaCoISDK ye2DI9mOCQWI75pH9azTJlA8eSOR6Xu0rGj9v9HGOIgIVhqh/GmeLrrJJlR6Tz2XXkp4 yMHwyypDxgMq9R3cQb5Q1qzBRmegRwxCMa7N7CHfZbN0rNoaNLG80aiPKnOq/ZqAOqKF nmTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717466771; x=1718071571; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=l2jOiTF4T3isqemcmHaO7N7kxplnAs9PyjvY121bHmg=; b=J6B4GDKZp5WYiM+TREsI3EEYdCN/I6vP5z5PEPJPU/WhSOd4Qjgy9B6lbFK+zGfdhC 2JF/Qu/cv6fS1tSCl7+1zTh9PXCEu+lqVHYKN9jmltriqnVsb53ecM4J8Qyvpdjk/mge feBh71WwxOfgk0OBXpCNE6ag2VsKG9+1LboxkaAdNzV34olh91rtb6bFK1uYLoyY0dnd u4KqS3AN2pljB2AOyLwDchZJeVh1sw4X8j2+xldkAdvnFZpFYDYhhn1ipQcM7QCnmDcQ yTzCh6cPccTPlvsYzoXMWukdcrHLkU4Z6BuloaMqFft11mxSe4NmQRoTnt/5qOCyHm4L BFzA== X-Forwarded-Encrypted: i=1; AJvYcCXjqAm9SwLZOLDvmI203R8xcWFT+1RuuLfQ6UzMk+JbK5xGB4wum3eISD7V8tTwm+GzLP3rSDTjWUcYbgyX2W1ZbAl5PtWURP14UWt+ X-Gm-Message-State: AOJu0YwS12HBFcfnMdF2iy1d/n9wnEcsnOgLqzFj/NyGeb0JeaEEZljk eW8iKK5+HHxJGM93o1GoA0dozenTVxc5+9qx6BYtc89YjYO1kKDPZOwD0Jxzm5WcqZ2Bkng7Bn2 zfuB9cg== X-Google-Smtp-Source: AGHT+IHLINdYJFspp1M7wxgddP2dalbHqDL0130GKKxjAHH18dLClBJpYOc4pWdHhcgiT8aRHRZymTgu/tgn X-Received: from yuanchu-desktop.svl.corp.google.com ([2620:15c:2a3:200:367f:7387:3dd2:73f1]) (user=yuanchu job=sendgmr) by 2002:a05:690c:7011:b0:627:e282:6630 with SMTP id 00721157ae682-62c79863eeamr30012097b3.10.1717466770573; Mon, 03 Jun 2024 19:06:10 -0700 (PDT) Date: Mon, 3 Jun 2024 19:05:43 -0700 In-Reply-To: <20240604020549.1017540-1-yuanchu@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240604020549.1017540-1-yuanchu@google.com> X-Mailer: git-send-email 2.45.1.467.gbab1589fc0-goog Message-ID: <20240604020549.1017540-3-yuanchu@google.com> Subject: [PATCH v2 2/8] mm: aggregate working set information into histograms From: Yuanchu Xie To: David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Muhammad Usama Anjum Cc: Kalesh Singh , Wei Xu , David Rientjes , Greg Kroah-Hartman , "Rafael J. Wysocki" , Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Shuah Khan , Yosry Ahmed , Matthew Wilcox , Sudarshan Rajagopalan , Kairui Song , "Michael S. Tsirkin" , Vasily Averin , Nhat Pham , Miaohe Lin , Qi Zheng , Abel Wu , "Vishal Moola (Oracle)" , Kefeng Wang , Yuanchu Xie , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Hierarchically aggregate all memcgs' MGLRU generations and their page counts into working set page age histograms. The histograms break down the system's working set per-node, per-anon/file. The sysfs interfaces are as follows: /sys/devices/system/node/nodeX/page_age A per-node page age histogram, showing an aggregate of the node's lruvecs. The information is extracted from MGLRU's per-generation page counters. Reading this file causes a hierarchical aging of all lruvecs, scanning pages and creates a new generation in each lruvec. For example: 1000 anon=3D0 file=3D0 2000 anon=3D0 file=3D0 100000 anon=3D5533696 file=3D5566464 18446744073709551615 anon=3D0 file=3D0 /sys/devices/system/node/nodeX/page_age_interval A comma separated list of time in milliseconds that configures what the page age histogram uses for aggregation. Signed-off-by: Yuanchu Xie --- drivers/base/node.c | 6 + include/linux/mmzone.h | 9 + include/linux/workingset_report.h | 79 ++++++ mm/Kconfig | 9 + mm/Makefile | 1 + mm/internal.h | 4 + mm/memcontrol.c | 2 + mm/mm_init.c | 2 + mm/mmzone.c | 2 + mm/vmscan.c | 10 +- mm/workingset_report.c | 451 ++++++++++++++++++++++++++++++ 11 files changed, 571 insertions(+), 4 deletions(-) create mode 100644 include/linux/workingset_report.h create mode 100644 mm/workingset_report.c diff --git a/drivers/base/node.c b/drivers/base/node.c index eb72580288e6..ba5b8720dbfa 100644 --- a/drivers/base/node.c +++ b/drivers/base/node.c @@ -20,6 +20,8 @@ #include #include #include +#include +#include =20 static const struct bus_type node_subsys =3D { .name =3D "node", @@ -626,6 +628,7 @@ static int register_node(struct node *node, int num) } else { hugetlb_register_node(node); compaction_register_node(node); + wsr_init_sysfs(node); } =20 return error; @@ -642,6 +645,9 @@ void unregister_node(struct node *node) { hugetlb_unregister_node(node); compaction_unregister_node(node); + wsr_remove_sysfs(node); + wsr_destroy_lruvec(mem_cgroup_lruvec(NULL, NODE_DATA(node->dev.id))); + wsr_destroy_pgdat(NODE_DATA(node->dev.id)); node_remove_accesses(node); node_remove_caches(node); device_unregister(&node->dev); diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 8f9c9590a42c..6aa84cd59152 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -24,6 +24,7 @@ #include #include #include +#include =20 /* Free memory management - zoned buddy allocator. */ #ifndef CONFIG_ARCH_FORCE_MAX_ORDER @@ -631,6 +632,9 @@ struct lruvec { struct lru_gen_mm_state mm_state; #endif #endif /* CONFIG_LRU_GEN */ +#ifdef CONFIG_WORKINGSET_REPORT + struct wsr_state wsr; +#endif /* CONFIG_WORKINGSET_REPORT */ #ifdef CONFIG_MEMCG struct pglist_data *pgdat; #endif @@ -1404,6 +1408,11 @@ typedef struct pglist_data { struct lru_gen_memcg memcg_lru; #endif =20 +#ifdef CONFIG_WORKINGSET_REPORT + struct mutex wsr_update_mutex; + struct wsr_report_bins __rcu *wsr_page_age_bins; +#endif + CACHELINE_PADDING(_pad2_); =20 /* Per-node vmstats */ diff --git a/include/linux/workingset_report.h b/include/linux/workingset_r= eport.h new file mode 100644 index 000000000000..d7c2ee14ec87 --- /dev/null +++ b/include/linux/workingset_report.h @@ -0,0 +1,79 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_WORKINGSET_REPORT_H +#define _LINUX_WORKINGSET_REPORT_H + +#include +#include + +struct mem_cgroup; +struct pglist_data; +struct node; +struct lruvec; + +#ifdef CONFIG_WORKINGSET_REPORT + +#define WORKINGSET_REPORT_MIN_NR_BINS 2 +#define WORKINGSET_REPORT_MAX_NR_BINS 32 + +#define WORKINGSET_INTERVAL_MAX ((unsigned long)-1) +#define ANON_AND_FILE 2 + +struct wsr_report_bin { + unsigned long idle_age; + unsigned long nr_pages[ANON_AND_FILE]; +}; + +struct wsr_report_bins { + /* excludes the WORKINGSET_INTERVAL_MAX bin */ + unsigned long nr_bins; + /* last bin contains WORKINGSET_INTERVAL_MAX */ + unsigned long idle_age[WORKINGSET_REPORT_MAX_NR_BINS]; + struct rcu_head rcu; +}; + +struct wsr_page_age_histo { + unsigned long timestamp; + struct wsr_report_bin bins[WORKINGSET_REPORT_MAX_NR_BINS]; +}; + +struct wsr_state { + /* breakdown of workingset by page age */ + struct mutex page_age_lock; + struct wsr_page_age_histo *page_age; +}; + +void wsr_init_lruvec(struct lruvec *lruvec); +void wsr_destroy_lruvec(struct lruvec *lruvec); +void wsr_init_pgdat(struct pglist_data *pgdat); +void wsr_destroy_pgdat(struct pglist_data *pgdat); +void wsr_init_sysfs(struct node *node); +void wsr_remove_sysfs(struct node *node); + +/* + * Returns true if the wsr is configured to be refreshed. + * The next refresh time is stored in refresh_time. + */ +bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, + struct pglist_data *pgdat); +#else +static inline void wsr_init_lruvec(struct lruvec *lruvec) +{ +} +static inline void wsr_destroy_lruvec(struct lruvec *lruvec) +{ +} +static inline void wsr_init_pgdat(struct pglist_data *pgdat) +{ +} +static inline void wsr_destroy_pgdat(struct pglist_data *pgdat) +{ +} +static inline void wsr_init_sysfs(struct node *node) +{ +} +static inline void wsr_remove_sysfs(struct node *node) +{ +} +#endif /* CONFIG_WORKINGSET_REPORT */ + +#endif /* _LINUX_WORKINGSET_REPORT_H */ diff --git a/mm/Kconfig b/mm/Kconfig index b4cb45255a54..03927ed2adbd 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1249,6 +1249,15 @@ config IOMMU_MM_DATA config EXECMEM bool =20 +config WORKINGSET_REPORT + bool "Working set reporting" + depends on LRU_GEN && SYSFS + help + Report system and per-memcg working set to userspace. + + This option exports stats and events giving the user more insight + into its memory working set. + source "mm/damon/Kconfig" =20 endmenu diff --git a/mm/Makefile b/mm/Makefile index 8fb85acda1b1..ed05af2bb3e3 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -96,6 +96,7 @@ obj-$(CONFIG_DEVICE_MIGRATION) +=3D migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) +=3D huge_memory.o khugepaged.o obj-$(CONFIG_PAGE_COUNTER) +=3D page_counter.o obj-$(CONFIG_MEMCG) +=3D memcontrol.o vmpressure.o +obj-$(CONFIG_WORKINGSET_REPORT) +=3D workingset_report.o ifdef CONFIG_SWAP obj-$(CONFIG_MEMCG) +=3D swap_cgroup.o endif diff --git a/mm/internal.h b/mm/internal.h index b2c75b12014e..b5cd86b3fec8 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -389,11 +389,15 @@ extern unsigned long highest_memmap_pfn; /* * in mm/vmscan.c: */ +struct scan_control; bool isolate_lru_page(struct page *page); bool folio_isolate_lru(struct folio *folio); void putback_lru_page(struct page *page); void folio_putback_lru(struct folio *folio); extern void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state = reason); +bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq, bool can= _swap, + bool force_scan); +void set_task_reclaim_state(struct task_struct *task, struct reclaim_state= *rs); =20 /* * in mm/rmap.c: diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7fad15b2290c..f973679e4a24 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -66,6 +66,7 @@ #include #include #include +#include #include "internal.h" #include #include @@ -5671,6 +5672,7 @@ static void free_mem_cgroup_per_node_info(struct mem_= cgroup *memcg, int node) if (!pn) return; =20 + wsr_destroy_lruvec(&pn->lruvec); free_percpu(pn->lruvec_stats_percpu); kfree(pn->lruvec_stats); kfree(pn); diff --git a/mm/mm_init.c b/mm/mm_init.c index f72b852bd5b8..5ec45e1b0f59 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -29,6 +29,7 @@ #include #include #include +#include #include "internal.h" #include "slab.h" #include "shuffle.h" @@ -1373,6 +1374,7 @@ static void __meminit pgdat_init_internals(struct pgl= ist_data *pgdat) =20 pgdat_page_ext_init(pgdat); lruvec_init(&pgdat->__lruvec); + wsr_init_pgdat(pgdat); } =20 static void __meminit zone_init_internals(struct zone *zone, enum zone_typ= e idx, int nid, diff --git a/mm/mmzone.c b/mm/mmzone.c index c01896eca736..477cd5ac1d78 100644 --- a/mm/mmzone.c +++ b/mm/mmzone.c @@ -90,6 +90,8 @@ void lruvec_init(struct lruvec *lruvec) */ list_del(&lruvec->lists[LRU_UNEVICTABLE]); =20 + wsr_init_lruvec(lruvec); + lru_gen_init_lruvec(lruvec); } =20 diff --git a/mm/vmscan.c b/mm/vmscan.c index 73f3718b33f7..a05f1e8e5cb3 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -56,6 +56,7 @@ #include #include #include +#include =20 #include #include @@ -250,8 +251,7 @@ static bool writeback_throttling_sane(struct scan_contr= ol *sc) } #endif =20 -static void set_task_reclaim_state(struct task_struct *task, - struct reclaim_state *rs) +void set_task_reclaim_state(struct task_struct *task, struct reclaim_state= *rs) { /* Check for an overwrite */ WARN_ON_ONCE(rs && task->reclaim_state); @@ -3842,8 +3842,8 @@ static bool inc_max_seq(struct lruvec *lruvec, unsign= ed long seq, return success; } =20 -static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq, - bool can_swap, bool force_scan) +bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq, bool can= _swap, + bool force_scan) { bool success; struct lru_gen_mm_walk *walk; @@ -5628,6 +5628,8 @@ static int __init init_lru_gen(void) if (sysfs_create_group(mm_kobj, &lru_gen_attr_group)) pr_err("lru_gen: failed to create sysfs group\n"); =20 + wsr_init_sysfs(NULL); + debugfs_create_file("lru_gen", 0644, NULL, NULL, &lru_gen_rw_fops); debugfs_create_file("lru_gen_full", 0444, NULL, NULL, &lru_gen_ro_fops); =20 diff --git a/mm/workingset_report.c b/mm/workingset_report.c new file mode 100644 index 000000000000..a4dcf62fcd96 --- /dev/null +++ b/mm/workingset_report.c @@ -0,0 +1,451 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "internal.h" + +void wsr_init_pgdat(struct pglist_data *pgdat) +{ + mutex_init(&pgdat->wsr_update_mutex); + RCU_INIT_POINTER(pgdat->wsr_page_age_bins, NULL); +} + +void wsr_destroy_pgdat(struct pglist_data *pgdat) +{ + struct wsr_report_bins __rcu *bins; + + mutex_lock(&pgdat->wsr_update_mutex); + bins =3D rcu_replace_pointer(pgdat->wsr_page_age_bins, NULL, + lockdep_is_held(&pgdat->wsr_update_mutex)); + kfree_rcu(bins, rcu); + mutex_unlock(&pgdat->wsr_update_mutex); + mutex_destroy(&pgdat->wsr_update_mutex); +} + +void wsr_init_lruvec(struct lruvec *lruvec) +{ + struct wsr_state *wsr =3D &lruvec->wsr; + + memset(wsr, 0, sizeof(*wsr)); + mutex_init(&wsr->page_age_lock); +} + +void wsr_destroy_lruvec(struct lruvec *lruvec) +{ + struct wsr_state *wsr =3D &lruvec->wsr; + + mutex_destroy(&wsr->page_age_lock); + kfree(wsr->page_age); + memset(wsr, 0, sizeof(*wsr)); +} + +static int workingset_report_intervals_parse(char *src, + struct wsr_report_bins *bins) +{ + int err =3D 0, i =3D 0; + char *cur, *next =3D strim(src); + + if (*next =3D=3D '\0') + return 0; + + while ((cur =3D strsep(&next, ","))) { + unsigned int interval; + + err =3D kstrtouint(cur, 0, &interval); + if (err) + goto out; + + bins->idle_age[i] =3D msecs_to_jiffies(interval); + if (i > 0 && bins->idle_age[i] <=3D bins->idle_age[i - 1]) { + err =3D -EINVAL; + goto out; + } + + if (++i =3D=3D WORKINGSET_REPORT_MAX_NR_BINS) { + err =3D -ERANGE; + goto out; + } + } + + if (i && i < WORKINGSET_REPORT_MIN_NR_BINS - 1) { + err =3D -ERANGE; + goto out; + } + + bins->nr_bins =3D i; + bins->idle_age[i] =3D WORKINGSET_INTERVAL_MAX; +out: + return err ?: i; +} + +static unsigned long get_gen_start_time(const struct lru_gen_folio *lrugen, + unsigned long seq, + unsigned long max_seq, + unsigned long curr_timestamp) +{ + int younger_gen; + + if (seq =3D=3D max_seq) + return curr_timestamp; + younger_gen =3D lru_gen_from_seq(seq + 1); + return READ_ONCE(lrugen->timestamps[younger_gen]); +} + +static void collect_page_age_type(const struct lru_gen_folio *lrugen, + struct wsr_report_bin *bin, + unsigned long max_seq, unsigned long min_seq, + unsigned long curr_timestamp, int type) +{ + unsigned long seq; + + for (seq =3D max_seq; seq + 1 > min_seq; seq--) { + int gen, zone; + unsigned long gen_end, gen_start, size =3D 0; + + gen =3D lru_gen_from_seq(seq); + + for (zone =3D 0; zone < MAX_NR_ZONES; zone++) + size +=3D max( + READ_ONCE(lrugen->nr_pages[gen][type][zone]), + 0L); + + gen_start =3D get_gen_start_time(lrugen, seq, max_seq, + curr_timestamp); + gen_end =3D READ_ONCE(lrugen->timestamps[gen]); + + while (bin->idle_age !=3D WORKINGSET_INTERVAL_MAX && + time_before(gen_end + bin->idle_age, curr_timestamp)) { + unsigned long gen_in_bin =3D (long)gen_start - + (long)curr_timestamp + + (long)bin->idle_age; + unsigned long gen_len =3D (long)gen_start - (long)gen_end; + + if (!gen_len) + break; + if (gen_in_bin) { + unsigned long split_bin =3D + size / gen_len * gen_in_bin; + + bin->nr_pages[type] +=3D split_bin; + size -=3D split_bin; + } + gen_start =3D curr_timestamp - bin->idle_age; + bin++; + } + bin->nr_pages[type] +=3D size; + } +} + +/* + * proportionally aggregate Multi-gen LRU bins into a working set report + * MGLRU generations: + * current time + * | max_seq timestamp + * | | max_seq - 1 timestamp + * | | | unbounded + * | | | | + * -------------------------------- + * | max_seq | ... | ... | min_seq + * -------------------------------- + * + * Bins: + * + * current time + * | current - idle_age[0] + * | | current - idle_age[1] + * | | | unbounded + * | | | | + * ------------------------------ + * | bin 0 | ... | ... | bin n-1 + * ------------------------------ + * + * Assume the heuristic that pages are in the MGLRU generation + * through uniform accesses, so we can aggregate them + * proportionally into bins. + */ +static void collect_page_age(struct wsr_page_age_histo *page_age, + const struct lruvec *lruvec) +{ + int type; + const struct lru_gen_folio *lrugen =3D &lruvec->lrugen; + unsigned long curr_timestamp =3D jiffies; + unsigned long max_seq =3D READ_ONCE((lruvec)->lrugen.max_seq); + unsigned long min_seq[ANON_AND_FILE] =3D { + READ_ONCE(lruvec->lrugen.min_seq[LRU_GEN_ANON]), + READ_ONCE(lruvec->lrugen.min_seq[LRU_GEN_FILE]), + }; + struct wsr_report_bin *bin =3D &page_age->bins[0]; + + for (type =3D 0; type < ANON_AND_FILE; type++) + collect_page_age_type(lrugen, bin, max_seq, min_seq[type], + curr_timestamp, type); +} + +/* First step: hierarchically scan child memcgs. */ +static void refresh_scan(struct wsr_state *wsr, struct mem_cgroup *root, + struct pglist_data *pgdat) +{ + struct mem_cgroup *memcg; + unsigned int flags; + struct reclaim_state rs =3D { 0 }; + + set_task_reclaim_state(current, &rs); + flags =3D memalloc_noreclaim_save(); + + memcg =3D mem_cgroup_iter(root, NULL, NULL); + do { + struct lruvec *lruvec =3D mem_cgroup_lruvec(memcg, pgdat); + unsigned long max_seq =3D READ_ONCE((lruvec)->lrugen.max_seq); + + /* + * setting can_swap=3Dtrue and force_scan=3Dtrue ensures + * proper workingset stats when the system cannot swap. + */ + try_to_inc_max_seq(lruvec, max_seq, true, true); + cond_resched(); + } while ((memcg =3D mem_cgroup_iter(root, memcg, NULL))); + + memalloc_noreclaim_restore(flags); + set_task_reclaim_state(current, NULL); +} + +/* Second step: aggregate child memcgs into the page age histogram. */ +static void refresh_aggregate(struct wsr_page_age_histo *page_age, + struct mem_cgroup *root, + struct pglist_data *pgdat) +{ + struct mem_cgroup *memcg; + struct wsr_report_bin *bin; + + for (bin =3D page_age->bins; + bin->idle_age !=3D WORKINGSET_INTERVAL_MAX; bin++) { + bin->nr_pages[0] =3D 0; + bin->nr_pages[1] =3D 0; + } + /* the last used bin has idle_age =3D=3D WORKINGSET_INTERVAL_MAX. */ + bin->nr_pages[0] =3D 0; + bin->nr_pages[1] =3D 0; + + memcg =3D mem_cgroup_iter(root, NULL, NULL); + do { + struct lruvec *lruvec =3D mem_cgroup_lruvec(memcg, pgdat); + + collect_page_age(page_age, lruvec); + cond_resched(); + } while ((memcg =3D mem_cgroup_iter(root, memcg, NULL))); + WRITE_ONCE(page_age->timestamp, jiffies); +} + +static void copy_node_bins(struct pglist_data *pgdat, + struct wsr_page_age_histo *page_age) +{ + struct wsr_report_bins *node_page_age_bins; + int i =3D 0; + + rcu_read_lock(); + node_page_age_bins =3D rcu_dereference(pgdat->wsr_page_age_bins); + if (!node_page_age_bins) + goto nocopy; + for (i =3D 0; i < node_page_age_bins->nr_bins; ++i) + page_age->bins[i].idle_age =3D node_page_age_bins->idle_age[i]; + +nocopy: + page_age->bins[i].idle_age =3D WORKINGSET_INTERVAL_MAX; + rcu_read_unlock(); +} + +bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, + struct pglist_data *pgdat) +{ + struct wsr_page_age_histo *page_age; + + if (!READ_ONCE(wsr->page_age)) + return false; + + refresh_scan(wsr, root, pgdat); + mutex_lock(&wsr->page_age_lock); + page_age =3D READ_ONCE(wsr->page_age); + if (page_age) { + copy_node_bins(pgdat, page_age); + refresh_aggregate(page_age, root, pgdat); + } + mutex_unlock(&wsr->page_age_lock); + return !!page_age; +} +EXPORT_SYMBOL_GPL(wsr_refresh_report); + +static struct pglist_data *kobj_to_pgdat(struct kobject *kobj) +{ + int nid =3D IS_ENABLED(CONFIG_NUMA) ? kobj_to_dev(kobj)->id : + first_memory_node; + + return NODE_DATA(nid); +} + +static struct wsr_state *kobj_to_wsr(struct kobject *kobj) +{ + return &mem_cgroup_lruvec(NULL, kobj_to_pgdat(kobj))->wsr; +} + +static ssize_t page_age_intervals_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct wsr_report_bins *bins; + int len =3D 0; + struct pglist_data *pgdat =3D kobj_to_pgdat(kobj); + + rcu_read_lock(); + bins =3D rcu_dereference(pgdat->wsr_page_age_bins); + if (bins) { + int i; + int nr_bins =3D bins->nr_bins; + + for (i =3D 0; i < bins->nr_bins; ++i) { + len +=3D sysfs_emit_at( + buf, len, "%u", + jiffies_to_msecs(bins->idle_age[i])); + if (i + 1 < nr_bins) + len +=3D sysfs_emit_at(buf, len, ","); + } + } + len +=3D sysfs_emit_at(buf, len, "\n"); + rcu_read_unlock(); + + return len; +} + +static ssize_t page_age_intervals_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *src, size_t len) +{ + struct wsr_report_bins *bins =3D NULL, __rcu *old; + char *buf =3D NULL; + int err =3D 0; + struct pglist_data *pgdat =3D kobj_to_pgdat(kobj); + + buf =3D kstrdup(src, GFP_KERNEL); + if (!buf) { + err =3D -ENOMEM; + goto failed; + } + + bins =3D + kzalloc(sizeof(struct wsr_report_bins), GFP_KERNEL); + + if (!bins) { + err =3D -ENOMEM; + goto failed; + } + + err =3D workingset_report_intervals_parse(buf, bins); + if (err < 0) + goto failed; + + if (err =3D=3D 0) { + kfree(bins); + bins =3D NULL; + } + + mutex_lock(&pgdat->wsr_update_mutex); + old =3D rcu_replace_pointer(pgdat->wsr_page_age_bins, bins, + lockdep_is_held(&pgdat->wsr_update_mutex)); + mutex_unlock(&pgdat->wsr_update_mutex); + kfree_rcu(old, rcu); + kfree(buf); + return len; +failed: + kfree(bins); + kfree(buf); + + return err; +} + +static struct kobj_attribute page_age_intervals_attr =3D + __ATTR_RW(page_age_intervals); + +static ssize_t page_age_show(struct kobject *kobj, struct kobj_attribute *= attr, + char *buf) +{ + struct wsr_report_bin *bin; + int ret =3D 0; + struct wsr_state *wsr =3D kobj_to_wsr(kobj); + + + mutex_lock(&wsr->page_age_lock); + if (!wsr->page_age) + wsr->page_age =3D + kzalloc(sizeof(struct wsr_page_age_histo), GFP_KERNEL); + mutex_unlock(&wsr->page_age_lock); + + wsr_refresh_report(wsr, NULL, kobj_to_pgdat(kobj)); + + mutex_lock(&wsr->page_age_lock); + if (!wsr->page_age) + goto unlock; + for (bin =3D wsr->page_age->bins; + bin->idle_age !=3D WORKINGSET_INTERVAL_MAX; bin++) + ret +=3D sysfs_emit_at(buf, ret, "%u anon=3D%lu file=3D%lu\n", + jiffies_to_msecs(bin->idle_age), + bin->nr_pages[0] * PAGE_SIZE, + bin->nr_pages[1] * PAGE_SIZE); + + ret +=3D sysfs_emit_at(buf, ret, "%lu anon=3D%lu file=3D%lu\n", + WORKINGSET_INTERVAL_MAX, + bin->nr_pages[0] * PAGE_SIZE, + bin->nr_pages[1] * PAGE_SIZE); + +unlock: + mutex_unlock(&wsr->page_age_lock); + return ret; +} + +static struct kobj_attribute page_age_attr =3D __ATTR_RO(page_age); + +static struct attribute *workingset_report_attrs[] =3D { + &page_age_intervals_attr.attr, &page_age_attr.attr, NULL +}; + +static const struct attribute_group workingset_report_attr_group =3D { + .name =3D "workingset_report", + .attrs =3D workingset_report_attrs, +}; + +void wsr_init_sysfs(struct node *node) +{ + struct kobject *kobj =3D node ? &node->dev.kobj : mm_kobj; + struct wsr_state *wsr; + + if (IS_ENABLED(CONFIG_NUMA) && !node) + return; + + wsr =3D kobj_to_wsr(kobj); + + if (sysfs_create_group(kobj, &workingset_report_attr_group)) + pr_warn("Workingset report failed to create sysfs files\n"); +} +EXPORT_SYMBOL_GPL(wsr_init_sysfs); + +void wsr_remove_sysfs(struct node *node) +{ + struct kobject *kobj =3D &node->dev.kobj; + struct wsr_state *wsr; + + if (IS_ENABLED(CONFIG_NUMA) && !node) + return; + + wsr =3D kobj_to_wsr(kobj); + sysfs_remove_group(kobj, &workingset_report_attr_group); +} +EXPORT_SYMBOL_GPL(wsr_remove_sysfs); --=20 2.45.1.467.gbab1589fc0-goog From nobody Fri Feb 13 01:40:26 2026 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3502613D89C for ; Tue, 4 Jun 2024 02:06:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717466774; cv=none; b=eL1q1v3z615DmG2CqxDZdRKLgi8f6kbWcg9Dc0xV0vqbbYshjQ/m+sbDKioqLLCMECh/BnlCYChdnRgfclXkGOFviZu/XQJPV95syyWKmiYoPev/dD2MvwwtFBss8gfC/NsECadGvyIZCGh56o2a27MM1Opl1fxaThT8grR8/V8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717466774; c=relaxed/simple; bh=fth2mh5hLwZQ+6IzfCTKsQwbyjH8dDmVn+u6o0NiQiY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Xz9Jvz7IBDSJzH6jExNCVMhCuvHorIpgXfKU0Qef366EmwR58mQPkulL3D8VR8Bo36hZ3Eqnue5gJNl0x8eMrVWipJIS1tbdv5pEOIZzCSqIi8ljm8HOdj6UVmHrGBy/qFotZt/IEQcoXKiHCtttz1ZhH/m1VsT9/Iql/rGPAB8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--yuanchu.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=jCspLKne; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--yuanchu.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="jCspLKne" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-df78efda253so8012825276.1 for ; Mon, 03 Jun 2024 19:06:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717466772; x=1718071572; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lTOg/v4agW4t4UIoB+YneZrty+w+zSP5tjy7abHeLhs=; b=jCspLKneDnH+idREYlO67FAi3+wE4vd8Jfq2IhxHwrPsqIS32F9zj/RbRdHvwxamwT Mi0CeLnfWoCrqrNWlvZ1FbotXhb8JVwsArJ8ypAibCmOC5pW6IAR6AwmqFOJkO6bQ7je QTSUVAHjetrJ+VzR1ybIQTlNfwQkGJnyWhP0e6sGn73CdoYv4yJD3UfhMgF82ShVD6DR HmsU+sThhR+ZPtpGy1+xF32KBn2TngHFbOZ11kS0hXI2rceqwfGd9+nmRmsBzQZakpGo gJCEnfQHa599ZqwogsTdStYQC7JCgtYvss9EwytDm1RDcPEaPNJp7qEHvR9uGCYbsNm/ aGjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717466772; x=1718071572; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lTOg/v4agW4t4UIoB+YneZrty+w+zSP5tjy7abHeLhs=; b=FKWt42B4HAb/a2swj6ZkFjfMGCrLzyisoap4XfOFzd599s1eSJanoJ7QqRI9hZPiR/ /aPJKDW/NAryu9IjcjPfj+06iR4Axjo+8MidVckJTOhr38C9grW0plNKYAofuNdZ9WAv NE68XfPq7TzQIeSjUH7l/GMUbUHbH56Nu+WhzOld32QUZkhS/DPwGW5UCe948Od/zrM0 j9GeIrVz2LG1R9aBk8wwOb3jSfILqkHCGtCFtv+y33sj9t1kut3xpwgofNvW4P/m6J0Z rjcW7jw32+qh5SIO4IkUPy4v/F1me4qHN1VZjGaHB+RnnimkiAoegXgZ77zUKR1lcnPj Pc6A== X-Forwarded-Encrypted: i=1; AJvYcCVqEN4jBcKLnm0cDr4v+uT7sefFbKHmzOx/87uSMeGKzbE0No5x/LEoLIi58zL1QQPYY5zVnetPN8u08/Yl36oo/AibidsM/HYACro6 X-Gm-Message-State: AOJu0YyOkY9FI/ILvFEQgDOztTSOGZGGbjJYS+EnKZXAr/zyagQoIhhd NlDHrWNY+J816WKTSruxBSLU5e2mx+dXW5X6xtsCR3dFNB4omGuI3XE5oFHGQ959vuczPpFj601 mh/gryg== X-Google-Smtp-Source: AGHT+IF3oIyaAWYZ8ktE5GFmyMb/f4z4VLJcbawoHVR7E/BZJtA687NFWcmcqqxVcMJp0b3YvGvnhH6eb1O6 X-Received: from yuanchu-desktop.svl.corp.google.com ([2620:15c:2a3:200:367f:7387:3dd2:73f1]) (user=yuanchu job=sendgmr) by 2002:a05:6902:2d42:b0:df7:83fa:2736 with SMTP id 3f1490d57ef6-dfa73dbc831mr913449276.11.1717466772327; Mon, 03 Jun 2024 19:06:12 -0700 (PDT) Date: Mon, 3 Jun 2024 19:05:44 -0700 In-Reply-To: <20240604020549.1017540-1-yuanchu@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240604020549.1017540-1-yuanchu@google.com> X-Mailer: git-send-email 2.45.1.467.gbab1589fc0-goog Message-ID: <20240604020549.1017540-4-yuanchu@google.com> Subject: [PATCH v2 3/8] mm: use refresh interval to rate-limit workingset report aggregation From: Yuanchu Xie To: David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Muhammad Usama Anjum Cc: Kalesh Singh , Wei Xu , David Rientjes , Greg Kroah-Hartman , "Rafael J. Wysocki" , Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Shuah Khan , Yosry Ahmed , Matthew Wilcox , Sudarshan Rajagopalan , Kairui Song , "Michael S. Tsirkin" , Vasily Averin , Nhat Pham , Miaohe Lin , Qi Zheng , Abel Wu , "Vishal Moola (Oracle)" , Kefeng Wang , Yuanchu Xie , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The refresh interval is a rate limiting factor to workingset page age histogram reads. When a workingset report is generated, a timestamp is noted, and the same report will be read until it expires beyond the refresh interval, at which point a new report is generated. Sysfs interface /sys/devices/system/node/nodeX/workingset_report/refresh_interval time in milliseconds specifying how long the report is valid for Signed-off-by: Yuanchu Xie --- include/linux/workingset_report.h | 1 + mm/workingset_report.c | 84 +++++++++++++++++++++++++------ 2 files changed, 70 insertions(+), 15 deletions(-) diff --git a/include/linux/workingset_report.h b/include/linux/workingset_r= eport.h index d7c2ee14ec87..8bae6a600410 100644 --- a/include/linux/workingset_report.h +++ b/include/linux/workingset_report.h @@ -37,6 +37,7 @@ struct wsr_page_age_histo { }; =20 struct wsr_state { + unsigned long refresh_interval; /* breakdown of workingset by page age */ struct mutex page_age_lock; struct wsr_page_age_histo *page_age; diff --git a/mm/workingset_report.c b/mm/workingset_report.c index a4dcf62fcd96..fe553c0a653e 100644 --- a/mm/workingset_report.c +++ b/mm/workingset_report.c @@ -195,7 +195,8 @@ static void collect_page_age(struct wsr_page_age_histo = *page_age, =20 /* First step: hierarchically scan child memcgs. */ static void refresh_scan(struct wsr_state *wsr, struct mem_cgroup *root, - struct pglist_data *pgdat) + struct pglist_data *pgdat, + unsigned long refresh_interval) { struct mem_cgroup *memcg; unsigned int flags; @@ -208,12 +209,15 @@ static void refresh_scan(struct wsr_state *wsr, struc= t mem_cgroup *root, do { struct lruvec *lruvec =3D mem_cgroup_lruvec(memcg, pgdat); unsigned long max_seq =3D READ_ONCE((lruvec)->lrugen.max_seq); + int gen =3D lru_gen_from_seq(max_seq); + unsigned long birth =3D READ_ONCE(lruvec->lrugen.timestamps[gen]); =20 /* * setting can_swap=3Dtrue and force_scan=3Dtrue ensures * proper workingset stats when the system cannot swap. */ - try_to_inc_max_seq(lruvec, max_seq, true, true); + if (time_is_before_jiffies(birth + refresh_interval)) + try_to_inc_max_seq(lruvec, max_seq, true, true); cond_resched(); } while ((memcg =3D mem_cgroup_iter(root, memcg, NULL))); =20 @@ -270,17 +274,25 @@ bool wsr_refresh_report(struct wsr_state *wsr, struct= mem_cgroup *root, struct pglist_data *pgdat) { struct wsr_page_age_histo *page_age; + unsigned long refresh_interval =3D READ_ONCE(wsr->refresh_interval); =20 if (!READ_ONCE(wsr->page_age)) return false; =20 - refresh_scan(wsr, root, pgdat); + if (!refresh_interval) + return false; + mutex_lock(&wsr->page_age_lock); page_age =3D READ_ONCE(wsr->page_age); - if (page_age) { - copy_node_bins(pgdat, page_age); - refresh_aggregate(page_age, root, pgdat); - } + if (!page_age) + goto unlock; + if (page_age->timestamp && + time_is_after_jiffies(page_age->timestamp + refresh_interval)) + goto unlock; + refresh_scan(wsr, root, pgdat, refresh_interval); + copy_node_bins(pgdat, page_age); + refresh_aggregate(page_age, root, pgdat); +unlock: mutex_unlock(&wsr->page_age_lock); return !!page_age; } @@ -299,6 +311,52 @@ static struct wsr_state *kobj_to_wsr(struct kobject *k= obj) return &mem_cgroup_lruvec(NULL, kobj_to_pgdat(kobj))->wsr; } =20 +static ssize_t refresh_interval_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct wsr_state *wsr =3D kobj_to_wsr(kobj); + unsigned int interval =3D READ_ONCE(wsr->refresh_interval); + + return sysfs_emit(buf, "%u\n", jiffies_to_msecs(interval)); +} + +static ssize_t refresh_interval_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t len) +{ + unsigned int interval; + int err; + struct wsr_state *wsr =3D kobj_to_wsr(kobj); + + err =3D kstrtouint(buf, 0, &interval); + if (err) + return err; + + mutex_lock(&wsr->page_age_lock); + if (interval && !wsr->page_age) { + struct wsr_page_age_histo *page_age =3D + kzalloc(sizeof(struct wsr_page_age_histo), GFP_KERNEL); + + if (!page_age) { + err =3D -ENOMEM; + goto unlock; + } + wsr->page_age =3D page_age; + } + if (!interval && wsr->page_age) { + kfree(wsr->page_age); + wsr->page_age =3D NULL; + } + + WRITE_ONCE(wsr->refresh_interval, msecs_to_jiffies(interval)); +unlock: + mutex_unlock(&wsr->page_age_lock); + return err ?: len; +} + +static struct kobj_attribute refresh_interval_attr =3D + __ATTR_RW(refresh_interval); + static ssize_t page_age_intervals_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf) { @@ -382,13 +440,6 @@ static ssize_t page_age_show(struct kobject *kobj, str= uct kobj_attribute *attr, int ret =3D 0; struct wsr_state *wsr =3D kobj_to_wsr(kobj); =20 - - mutex_lock(&wsr->page_age_lock); - if (!wsr->page_age) - wsr->page_age =3D - kzalloc(sizeof(struct wsr_page_age_histo), GFP_KERNEL); - mutex_unlock(&wsr->page_age_lock); - wsr_refresh_report(wsr, NULL, kobj_to_pgdat(kobj)); =20 mutex_lock(&wsr->page_age_lock); @@ -414,7 +465,10 @@ static ssize_t page_age_show(struct kobject *kobj, str= uct kobj_attribute *attr, static struct kobj_attribute page_age_attr =3D __ATTR_RO(page_age); =20 static struct attribute *workingset_report_attrs[] =3D { - &page_age_intervals_attr.attr, &page_age_attr.attr, NULL + &refresh_interval_attr.attr, + &page_age_intervals_attr.attr, + &page_age_attr.attr, + NULL }; =20 static const struct attribute_group workingset_report_attr_group =3D { --=20 2.45.1.467.gbab1589fc0-goog From nobody Fri Feb 13 01:40:26 2026 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 45B3113DDD1 for ; Tue, 4 Jun 2024 02:06:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717466778; cv=none; b=XccZyO1TQAJ97TKBcvsNZ+ky5ZzvFnBazf0D7zygmApz4eCR0SzOckAwklUCHJOsqHM/cIxvcnsYqtP0BA0OVZR8E5lwvIac18WPL2/TTh1C53RL/9mvToCiaUEuM03c4IUEF5aSSY6e6bFDpDEj6FDgs/KNqtnW+Z0Ab9tPulA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717466778; c=relaxed/simple; bh=jMjcWKQ200vzGtXDYHEO2KK7tPtckVQowNHX8THkLrs=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=oy8T/VQ9/bCWigrpBPH1adX5bq7dJZRj5Ap1maKYW/0ZLn400Uu1hAwCNVEpdnJESPSkx1r9it8x+f/CAq56NTUFaJvbWD0PhgKhdfQEmPo5HGCpaT8ioMFASU3+wgQASlht27P/eVYQeqB3KNBp0Vwk1cxpZ/PPFpWmzgEzyTA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--yuanchu.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Xat/sStf; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--yuanchu.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Xat/sStf" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-df77196089eso7737323276.1 for ; Mon, 03 Jun 2024 19:06:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717466774; x=1718071574; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=pSleViApJ1HtHahn3WgRlaq3aZcfr2+vjvy3KUqVlAc=; b=Xat/sStfTYxl9v9TuSfddh62UrF0ql6ra1Y/4qofyqgNbbnR6W0n7gff3MRNxYquSc XgL2UvlPXZCbnOgufv9eRp1HpgJzv1ftlWbTJukehCxa3M5siFBJt7F6XmZwdc5qiAAD sXk3cMy8yFfGBBhmrVF2xaR8n9KbcDjq4TFDtBSigE46ZFwY1PawMsYVfKkT12Fu/dCF RT33ccriz6KF6N6mdwTDRZU4XV1XMF16B8jOpRT60ISkI/usKZ4qbtQoZcgCi/t23z7P fF1ZgxgeJIezzKJX4DMXaIWN7uY1uAqCKSewRVzZgxEhTrQMEcH/ImHmM/qPLR/8Us/v 627g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717466774; x=1718071574; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pSleViApJ1HtHahn3WgRlaq3aZcfr2+vjvy3KUqVlAc=; b=qgFfYOzwCyUzdwiBAtPK0vu4jOPkQs+F7K+Tgk25LNd/ZrKW7iUXFV7I1m0kE44xR5 QOk2RBFwG0BOBtTZAqCcMi+Us7kfVsr/4ceKmT6gAyUUzd3pTHAXDFzxHdpjTLLXV3c6 DcBrWyAHtjXwx2KZDkZB5tggO7fs4uUc54ofo+Gi8hsQYQi5MyIB9QuZqB+N+7r4gE5I fhgcMmPj5AJUDO9fGZXHEerW3gkYMAEAlXux5XHP4rT/Kb/D9FUojdN+3pU+2pDkkswv aGEjlC1biwn1fO1fntOB3bDAq508sW2Yv7eMaX8C8fh2XGKFFWbSFg11MowymyGVrsuX 9KpA== X-Forwarded-Encrypted: i=1; AJvYcCUfzxTCBsKCqESEJB7O1d3ZM5HzIUWvtgiWsMsG7g2VhWOcObriCztfGcjlgSqNseKU5afD2IPThSoeYUwWRT4xbMDCwPcFO1z1qKIu X-Gm-Message-State: AOJu0Yz7h5jzCWt9c0b6uOJpFbsmuQn/HRSXpBQOTQfciTw0IZzgjb3v Exhro/KnsKG7Kq9puhpxRCB+PpAX++GFtteCT58sN7pkLYO0QYBECa+gAMsEGCBhbNM9Dl+9SMy 9CmQ2Yw== X-Google-Smtp-Source: AGHT+IHvvDuGpbRREo0VJ2AwDl8RRdoBG5TFOYaDueQhzCeuZnnIw7idyIAR3QbgubkDUEdsnEZqQs0sZPOl X-Received: from yuanchu-desktop.svl.corp.google.com ([2620:15c:2a3:200:367f:7387:3dd2:73f1]) (user=yuanchu job=sendgmr) by 2002:a05:6902:2b03:b0:dfa:4b1f:56f6 with SMTP id 3f1490d57ef6-dfa73bf78b3mr1338312276.5.1717466774166; Mon, 03 Jun 2024 19:06:14 -0700 (PDT) Date: Mon, 3 Jun 2024 19:05:45 -0700 In-Reply-To: <20240604020549.1017540-1-yuanchu@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240604020549.1017540-1-yuanchu@google.com> X-Mailer: git-send-email 2.45.1.467.gbab1589fc0-goog Message-ID: <20240604020549.1017540-5-yuanchu@google.com> Subject: [PATCH v2 4/8] mm: report workingset during memory pressure driven scanning From: Yuanchu Xie To: David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Muhammad Usama Anjum Cc: Kalesh Singh , Wei Xu , David Rientjes , Greg Kroah-Hartman , "Rafael J. Wysocki" , Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Shuah Khan , Yosry Ahmed , Matthew Wilcox , Sudarshan Rajagopalan , Kairui Song , "Michael S. Tsirkin" , Vasily Averin , Nhat Pham , Miaohe Lin , Qi Zheng , Abel Wu , "Vishal Moola (Oracle)" , Kefeng Wang , Yuanchu Xie , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When a node reaches its low watermarks and wakes up kswapd, notify all userspace programs waiting on the workingset page age histogram of the memory pressure, so a userspace agent can read the workingset report in time and make policy decisions, such as logging, oom-killing, or migration. Sysfs interface: /sys/devices/system/node/nodeX/workingset_report/report_threshold time in milliseconds that specifies how often the userspace agent can be notified for node memory pressure. Signed-off-by: Yuanchu Xie --- include/linux/workingset_report.h | 4 +++ mm/internal.h | 12 ++++++++ mm/vmscan.c | 46 +++++++++++++++++++++++++++++++ mm/workingset_report.c | 43 ++++++++++++++++++++++++++++- 4 files changed, 104 insertions(+), 1 deletion(-) diff --git a/include/linux/workingset_report.h b/include/linux/workingset_r= eport.h index 8bae6a600410..2ec8b927b200 100644 --- a/include/linux/workingset_report.h +++ b/include/linux/workingset_report.h @@ -37,7 +37,11 @@ struct wsr_page_age_histo { }; =20 struct wsr_state { + unsigned long report_threshold; unsigned long refresh_interval; + + struct kernfs_node *page_age_sys_file; + /* breakdown of workingset by page age */ struct mutex page_age_lock; struct wsr_page_age_histo *page_age; diff --git a/mm/internal.h b/mm/internal.h index b5cd86b3fec8..3246384317f6 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -399,6 +399,18 @@ bool try_to_inc_max_seq(struct lruvec *lruvec, unsigne= d long seq, bool can_swap, bool force_scan); void set_task_reclaim_state(struct task_struct *task, struct reclaim_state= *rs); =20 +#ifdef CONFIG_WORKINGSET_REPORT +/* + * in mm/wsr.c + */ +void notify_workingset(struct mem_cgroup *memcg, struct pglist_data *pgdat= ); +#else +static inline void notify_workingset(struct mem_cgroup *memcg, + struct pglist_data *pgdat) +{ +} +#endif + /* * in mm/rmap.c: */ diff --git a/mm/vmscan.c b/mm/vmscan.c index a05f1e8e5cb3..9bba7c05c128 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2559,6 +2559,15 @@ static bool can_age_anon_pages(struct pglist_data *p= gdat, return can_demote(pgdat->node_id, sc); } =20 +#ifdef CONFIG_WORKINGSET_REPORT +static void try_to_report_workingset(struct pglist_data *pgdat, struct sca= n_control *sc); +#else +static inline void try_to_report_workingset(struct pglist_data *pgdat, + struct scan_control *sc) +{ +} +#endif + #ifdef CONFIG_LRU_GEN =20 #ifdef CONFIG_LRU_GEN_ENABLED @@ -3962,6 +3971,8 @@ static void lru_gen_age_node(struct pglist_data *pgda= t, struct scan_control *sc) if (!min_ttl || sc->order || sc->priority =3D=3D DEF_PRIORITY) return; =20 + try_to_report_workingset(pgdat, sc); + memcg =3D mem_cgroup_iter(NULL, NULL, NULL); do { struct lruvec *lruvec =3D mem_cgroup_lruvec(memcg, pgdat); @@ -5637,6 +5648,38 @@ static int __init init_lru_gen(void) }; late_initcall(init_lru_gen); =20 +#ifdef CONFIG_WORKINGSET_REPORT +static void try_to_report_workingset(struct pglist_data *pgdat, + struct scan_control *sc) +{ + struct mem_cgroup *memcg =3D sc->target_mem_cgroup; + struct wsr_state *wsr =3D &mem_cgroup_lruvec(memcg, pgdat)->wsr; + unsigned long threshold =3D READ_ONCE(wsr->report_threshold); + + if (sc->priority =3D=3D DEF_PRIORITY) + return; + + if (!threshold) + return; + + if (!mutex_trylock(&wsr->page_age_lock)) + return; + + if (!wsr->page_age) { + mutex_unlock(&wsr->page_age_lock); + return; + } + + if (time_is_after_jiffies(wsr->page_age->timestamp + threshold)) { + mutex_unlock(&wsr->page_age_lock); + return; + } + + mutex_unlock(&wsr->page_age_lock); + notify_workingset(memcg, pgdat); +} +#endif /* CONFIG_WORKINGSET_REPORT */ + #else /* !CONFIG_LRU_GEN */ =20 static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_contro= l *sc) @@ -6167,6 +6210,9 @@ static void shrink_zones(struct zonelist *zonelist, s= truct scan_control *sc) if (zone->zone_pgdat =3D=3D last_pgdat) continue; last_pgdat =3D zone->zone_pgdat; + + if (!sc->proactive) + try_to_report_workingset(zone->zone_pgdat, sc); shrink_node(zone->zone_pgdat, sc); } =20 diff --git a/mm/workingset_report.c b/mm/workingset_report.c index fe553c0a653e..801ac8e5c1da 100644 --- a/mm/workingset_report.c +++ b/mm/workingset_report.c @@ -311,6 +311,33 @@ static struct wsr_state *kobj_to_wsr(struct kobject *k= obj) return &mem_cgroup_lruvec(NULL, kobj_to_pgdat(kobj))->wsr; } =20 +static ssize_t report_threshold_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct wsr_state *wsr =3D kobj_to_wsr(kobj); + unsigned int threshold =3D READ_ONCE(wsr->report_threshold); + + return sysfs_emit(buf, "%u\n", jiffies_to_msecs(threshold)); +} + +static ssize_t report_threshold_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t len) +{ + unsigned int threshold; + struct wsr_state *wsr =3D kobj_to_wsr(kobj); + + if (kstrtouint(buf, 0, &threshold)) + return -EINVAL; + + WRITE_ONCE(wsr->report_threshold, msecs_to_jiffies(threshold)); + + return len; +} + +static struct kobj_attribute report_threshold_attr =3D + __ATTR_RW(report_threshold); + static ssize_t refresh_interval_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf) { @@ -465,6 +492,7 @@ static ssize_t page_age_show(struct kobject *kobj, stru= ct kobj_attribute *attr, static struct kobj_attribute page_age_attr =3D __ATTR_RO(page_age); =20 static struct attribute *workingset_report_attrs[] =3D { + &report_threshold_attr.attr, &refresh_interval_attr.attr, &page_age_intervals_attr.attr, &page_age_attr.attr, @@ -486,8 +514,13 @@ void wsr_init_sysfs(struct node *node) =20 wsr =3D kobj_to_wsr(kobj); =20 - if (sysfs_create_group(kobj, &workingset_report_attr_group)) + if (sysfs_create_group(kobj, &workingset_report_attr_group)) { pr_warn("Workingset report failed to create sysfs files\n"); + return; + } + + wsr->page_age_sys_file =3D + kernfs_walk_and_get(kobj->sd, "workingset_report/page_age"); } EXPORT_SYMBOL_GPL(wsr_init_sysfs); =20 @@ -500,6 +533,14 @@ void wsr_remove_sysfs(struct node *node) return; =20 wsr =3D kobj_to_wsr(kobj); + kernfs_put(wsr->page_age_sys_file); sysfs_remove_group(kobj, &workingset_report_attr_group); } EXPORT_SYMBOL_GPL(wsr_remove_sysfs); + +void notify_workingset(struct mem_cgroup *memcg, struct pglist_data *pgdat) +{ + struct wsr_state *wsr =3D &mem_cgroup_lruvec(memcg, pgdat)->wsr; + + kernfs_notify(wsr->page_age_sys_file); +} --=20 2.45.1.467.gbab1589fc0-goog From nobody Fri Feb 13 01:40:26 2026 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 39E4013DDBA for ; Tue, 4 Jun 2024 02:06:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717466779; cv=none; b=BYr5FKGXqKcWq1DbvH/zrCasBahfP5sgExbNI6Wl/B0r6QU+sr2J4iYz2jboAGUlgTuoLBeiw2vUuc+v6IV3zXGaiKoxM8WNbHQkaHnXiuIIWZt2l7MHD9jQAogUqlbTssKDUbj/QGNTmh5qu+eNIgOnL07nnkD5t7/sjp8QE14= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717466779; c=relaxed/simple; bh=WzqkPugbOiVbWg+Bz3/39mPsvZeHjQJkUc7aNbm+ez4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=QMxMf3sAc+ERjQdXnbNI9TF/TqESYtRqNjUUDIcjXNeqcpARKTZ2jPNrzgDrzpHJ3oYUW4h93b4gW9CVNIYwjSLGwXQTeVkWKxdJcuE1F3BqTD3v5DHyT5CAwKQ1P6JvMG6rczKnW9Q8Bc0cB6x5jcaNvE1iIWVPNTQDINYG+AI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--yuanchu.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=frO1Vxxv; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--yuanchu.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="frO1Vxxv" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-627f20cff42so83743087b3.0 for ; Mon, 03 Jun 2024 19:06:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717466776; x=1718071576; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=dRO4mHrw3JGh6nKhs6HXawOt4KQi22Ngkfo+vgcJs3s=; b=frO1VxxvzOztPw+gmxtFxjLugPMt7kPdrfiC5gbHQEKhke8UM0waezUQ8uiIxbZIvQ A2sR5dzsJ/eRqFThahsNDdm2cCH7YEheOsgdFhaaylfm+7sYTy02pwlVXCEmvEa4vg9V 4k/AVLRzgUzFG+Pp6wUNOMjLg09unZ7JSZhv2xVEF8oMucfgda5Qh8SC3Dwpe73RgUHy Zh7pjh7lQ9JsxpjS+w/2uqE+t7Z3Sy6dseD678ijc8yo3YmQQGZcnChtY+M2R9nfndeV n7v8QqRLpYPrs9CnzIM0UPL0qvixINQ2E2v8EUWcxdqFwXff7RoYTIVRkk0RNiSzTAQ6 dOUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717466776; x=1718071576; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dRO4mHrw3JGh6nKhs6HXawOt4KQi22Ngkfo+vgcJs3s=; b=sIqOPKp9UC8qR23tF0oXcPevZWeBI0YpyhWcyHAgWPd6sLSG2JRE+EYokIjhMX+Vhz 9yFp/dhgkcIVpWxXTM71Ujmte/N2EHSuVGAOZZTcYe+baJko1MX9CAsLew3J44f04YaD XGq0i9u4wGFpiHv/H/R5BloGrmog8liA8x6Uti5vrFhPHuZHDoasgYIjMEcLtJf17bVD Pe/SygJ9T/pRGXOawiwfWeN4dAn831hGUWR3IlXQagJhsLc4O40ItcbyLq9ZCze3YR7h BgvAJebQdx/RWAOTSK+KCKUTzn48Cd6yKWSMpPBre4o5y59aERqoqNJCROW6mTECQtSH noVQ== X-Forwarded-Encrypted: i=1; AJvYcCVpMO5a+27ym2uMHQh2PGKVyRD4GaFGJkdqVdT4Ko9AoRXDD/8+nujg/pE2FglwrHIGDOIjeivEEvdUfsOzy2oyWfCJqiKkHG8DMAba X-Gm-Message-State: AOJu0YzbSeCAaag2pztyGnIvbHgOdiLxUfgvCY/3un+zbd6/XLTsI27w KJkxP1RBYfucPNgAtlJGOVmxqHVGtiAy89Z/zG+F58HKen5u6JfUBfc9xgfzvvz+jvEPJyXIQSJ Plnm7WQ== X-Google-Smtp-Source: AGHT+IEA7IxV4E1j3KloZIYFfUw/fULB2Z+7pThTKaRON0/8bQkhXcDdKIm+z8+ZQ9n1pMUXwDRfYO5MgVc/ X-Received: from yuanchu-desktop.svl.corp.google.com ([2620:15c:2a3:200:367f:7387:3dd2:73f1]) (user=yuanchu job=sendgmr) by 2002:a05:690c:fd5:b0:61c:89a4:dd5f with SMTP id 00721157ae682-62c794a2504mr34231327b3.0.1717466776247; Mon, 03 Jun 2024 19:06:16 -0700 (PDT) Date: Mon, 3 Jun 2024 19:05:46 -0700 In-Reply-To: <20240604020549.1017540-1-yuanchu@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240604020549.1017540-1-yuanchu@google.com> X-Mailer: git-send-email 2.45.1.467.gbab1589fc0-goog Message-ID: <20240604020549.1017540-6-yuanchu@google.com> Subject: [PATCH v2 5/8] mm: extend working set reporting to memcgs From: Yuanchu Xie To: David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Muhammad Usama Anjum Cc: Kalesh Singh , Wei Xu , David Rientjes , Greg Kroah-Hartman , "Rafael J. Wysocki" , Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Shuah Khan , Yosry Ahmed , Matthew Wilcox , Sudarshan Rajagopalan , Kairui Song , "Michael S. Tsirkin" , Vasily Averin , Nhat Pham , Miaohe Lin , Qi Zheng , Abel Wu , "Vishal Moola (Oracle)" , Kefeng Wang , Yuanchu Xie , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Break down the system-wide working set reporting into per-memcg reports, which aggregages its children hierarchically. The per-node working set reporting histograms and refresh/report threshold files are presented as memcg files, showing a report containing all the nodes. The per-node page age interval is configurable in sysfs and not available per-memcg, while the refresh interval and report threshold are configured per-memcg. Memcg interface: /sys/fs/cgroup/.../memory.workingset.page_age The memcg equivalent of the sysfs workingset page age histogram, breaks down the workingset of this memcg and its children into page age intervals. Each node is prefixed with a node header and a newline. Non-proactive direct reclaim on this memcg can also wake up userspace agents that are waiting on this file. e.g. N0 1000 anon=3D0 file=3D0 2000 anon=3D0 file=3D0 3000 anon=3D0 file=3D0 4000 anon=3D0 file=3D0 5000 anon=3D0 file=3D0 18446744073709551615 anon=3D0 file=3D0 /sys/fs/cgroup/.../memory.workingset.refresh_interval The memcg equivalent of the sysfs refresh interval. A per-node number of how much time a page age histogram is valid for, in milliseconds. e.g. echo N0=3D2000 > memory.workingset.refresh_interval /sys/fs/cgroup/.../memory.workingset.report_threshold The memcg equivalent of the sysfs report threshold. A per-node number of how often userspace agent waiting on the page age histogram can be woken up, in milliseconds. e.g. echo N0=3D1000 > memory.workingset.report_threshold Signed-off-by: Yuanchu Xie --- include/linux/memcontrol.h | 5 + include/linux/workingset_report.h | 6 +- mm/internal.h | 2 + mm/memcontrol.c | 178 +++++++++++++++++++++++++++++- mm/workingset_report.c | 12 +- 5 files changed, 198 insertions(+), 5 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 030d34e9d117..91b08123950b 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -319,6 +319,11 @@ struct mem_cgroup { struct lru_gen_mm_list mm_list; #endif =20 +#ifdef CONFIG_WORKINGSET_REPORT + /* memory.workingset.page_age file */ + struct cgroup_file workingset_page_age_file; +#endif + struct mem_cgroup_per_node *nodeinfo[]; }; =20 diff --git a/include/linux/workingset_report.h b/include/linux/workingset_r= eport.h index 2ec8b927b200..ae412d408037 100644 --- a/include/linux/workingset_report.h +++ b/include/linux/workingset_report.h @@ -9,6 +9,7 @@ struct mem_cgroup; struct pglist_data; struct node; struct lruvec; +struct cgroup_file; =20 #ifdef CONFIG_WORKINGSET_REPORT =20 @@ -40,7 +41,10 @@ struct wsr_state { unsigned long report_threshold; unsigned long refresh_interval; =20 - struct kernfs_node *page_age_sys_file; + union { + struct kernfs_node *page_age_sys_file; + struct cgroup_file *page_age_cgroup_file; + }; =20 /* breakdown of workingset by page age */ struct mutex page_age_lock; diff --git a/mm/internal.h b/mm/internal.h index 3246384317f6..cf523a7c2048 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -404,6 +404,8 @@ void set_task_reclaim_state(struct task_struct *task, s= truct reclaim_state *rs); * in mm/wsr.c */ void notify_workingset(struct mem_cgroup *memcg, struct pglist_data *pgdat= ); +int workingset_report_intervals_parse(char *src, + struct wsr_report_bins *bins); #else static inline void notify_workingset(struct mem_cgroup *memcg, struct pglist_data *pgdat) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f973679e4a24..48cdc9422794 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -7226,6 +7226,162 @@ static ssize_t memory_reclaim(struct kernfs_open_fi= le *of, char *buf, return nbytes; } =20 +#ifdef CONFIG_WORKINGSET_REPORT +static int memory_ws_refresh_interval_show(struct seq_file *m, void *v) +{ + int nid; + struct mem_cgroup *memcg =3D mem_cgroup_from_seq(m); + + for_each_node_state(nid, N_MEMORY) { + struct wsr_state *wsr =3D + &mem_cgroup_lruvec(memcg, NODE_DATA(nid))->wsr; + + seq_printf(m, "N%d=3D%u ", nid, + jiffies_to_msecs(READ_ONCE(wsr->refresh_interval))); + } + seq_putc(m, '\n'); + + return 0; +} + +static ssize_t memory_wsr_threshold_parse(char *buf, size_t nbytes, + unsigned int *nid_out, + unsigned int *msecs) +{ + char *node, *threshold; + unsigned int nid; + int err; + + buf =3D strstrip(buf); + threshold =3D buf; + node =3D strsep(&threshold, "=3D"); + + if (*node !=3D 'N') + return -EINVAL; + + err =3D kstrtouint(node + 1, 0, &nid); + if (err) + return err; + + if (nid >=3D nr_node_ids || !node_state(nid, N_MEMORY)) + return -EINVAL; + + err =3D kstrtouint(threshold, 0, msecs); + if (err) + return err; + + *nid_out =3D nid; + + return nbytes; +} + +static ssize_t memory_ws_refresh_interval_write(struct kernfs_open_file *o= f, + char *buf, size_t nbytes, + loff_t off) +{ + unsigned int nid, msecs; + struct wsr_state *wsr; + struct mem_cgroup *memcg =3D mem_cgroup_from_css(of_css(of)); + ssize_t ret =3D memory_wsr_threshold_parse(buf, nbytes, &nid, &msecs); + + if (ret < 0) + return ret; + + wsr =3D &mem_cgroup_lruvec(memcg, NODE_DATA(nid))->wsr; + + mutex_lock(&wsr->page_age_lock); + if (msecs && !wsr->page_age) { + struct wsr_page_age_histo *page_age =3D + kzalloc(sizeof(struct wsr_page_age_histo), GFP_KERNEL); + + if (!page_age) { + ret =3D -ENOMEM; + goto unlock; + } + wsr->page_age =3D page_age; + } + if (!msecs && wsr->page_age) { + kfree(wsr->page_age); + wsr->page_age =3D NULL; + } + + WRITE_ONCE(wsr->refresh_interval, msecs_to_jiffies(msecs)); +unlock: + mutex_unlock(&wsr->page_age_lock); + return ret; +} + +static int memory_ws_report_threshold_show(struct seq_file *m, void *v) +{ + int nid; + struct mem_cgroup *memcg =3D mem_cgroup_from_seq(m); + + for_each_node_state(nid, N_MEMORY) { + struct wsr_state *wsr =3D + &mem_cgroup_lruvec(memcg, NODE_DATA(nid))->wsr; + + seq_printf(m, "N%d=3D%u ", nid, + jiffies_to_msecs(READ_ONCE(wsr->report_threshold))); + } + seq_putc(m, '\n'); + + return 0; +} + +static ssize_t memory_ws_report_threshold_write(struct kernfs_open_file *o= f, + char *buf, size_t nbytes, + loff_t off) +{ + unsigned int nid, msecs; + struct wsr_state *wsr; + struct mem_cgroup *memcg =3D mem_cgroup_from_css(of_css(of)); + ssize_t ret =3D memory_wsr_threshold_parse(buf, nbytes, &nid, &msecs); + + if (ret < 0) + return ret; + + wsr =3D &mem_cgroup_lruvec(memcg, NODE_DATA(nid))->wsr; + WRITE_ONCE(wsr->report_threshold, msecs_to_jiffies(msecs)); + return ret; +} + +static int memory_ws_page_age_show(struct seq_file *m, void *v) +{ + int nid; + struct mem_cgroup *memcg =3D mem_cgroup_from_seq(m); + + for_each_node_state(nid, N_MEMORY) { + struct wsr_state *wsr =3D + &mem_cgroup_lruvec(memcg, NODE_DATA(nid))->wsr; + struct wsr_report_bin *bin; + + if (!READ_ONCE(wsr->page_age)) + continue; + + wsr_refresh_report(wsr, memcg, NODE_DATA(nid)); + mutex_lock(&wsr->page_age_lock); + if (!wsr->page_age) + goto unlock; + seq_printf(m, "N%d\n", nid); + for (bin =3D wsr->page_age->bins; + bin->idle_age !=3D WORKINGSET_INTERVAL_MAX; bin++) + seq_printf(m, "%u anon=3D%lu file=3D%lu\n", + jiffies_to_msecs(bin->idle_age), + bin->nr_pages[0] * PAGE_SIZE, + bin->nr_pages[1] * PAGE_SIZE); + + seq_printf(m, "%lu anon=3D%lu file=3D%lu\n", WORKINGSET_INTERVAL_MAX, + bin->nr_pages[0] * PAGE_SIZE, + bin->nr_pages[1] * PAGE_SIZE); + +unlock: + mutex_unlock(&wsr->page_age_lock); + } + + return 0; +} +#endif + static struct cftype memory_files[] =3D { { .name =3D "current", @@ -7294,7 +7450,27 @@ static struct cftype memory_files[] =3D { .flags =3D CFTYPE_NS_DELEGATABLE, .write =3D memory_reclaim, }, - { } /* terminate */ +#ifdef CONFIG_WORKINGSET_REPORT + { + .name =3D "workingset.refresh_interval", + .flags =3D CFTYPE_NOT_ON_ROOT | CFTYPE_NS_DELEGATABLE, + .seq_show =3D memory_ws_refresh_interval_show, + .write =3D memory_ws_refresh_interval_write, + }, + { + .name =3D "workingset.report_threshold", + .flags =3D CFTYPE_NOT_ON_ROOT | CFTYPE_NS_DELEGATABLE, + .seq_show =3D memory_ws_report_threshold_show, + .write =3D memory_ws_report_threshold_write, + }, + { + .name =3D "workingset.page_age", + .flags =3D CFTYPE_NOT_ON_ROOT | CFTYPE_NS_DELEGATABLE, + .file_offset =3D offsetof(struct mem_cgroup, workingset_page_age_file), + .seq_show =3D memory_ws_page_age_show, + }, +#endif + {} /* terminate */ }; =20 struct cgroup_subsys memory_cgrp_subsys =3D { diff --git a/mm/workingset_report.c b/mm/workingset_report.c index 801ac8e5c1da..72f2cad85a0d 100644 --- a/mm/workingset_report.c +++ b/mm/workingset_report.c @@ -37,9 +37,12 @@ void wsr_destroy_pgdat(struct pglist_data *pgdat) void wsr_init_lruvec(struct lruvec *lruvec) { struct wsr_state *wsr =3D &lruvec->wsr; + struct mem_cgroup *memcg =3D lruvec_memcg(lruvec); =20 memset(wsr, 0, sizeof(*wsr)); mutex_init(&wsr->page_age_lock); + if (memcg && !mem_cgroup_is_root(memcg)) + wsr->page_age_cgroup_file =3D &memcg->workingset_page_age_file; } =20 void wsr_destroy_lruvec(struct lruvec *lruvec) @@ -51,8 +54,8 @@ void wsr_destroy_lruvec(struct lruvec *lruvec) memset(wsr, 0, sizeof(*wsr)); } =20 -static int workingset_report_intervals_parse(char *src, - struct wsr_report_bins *bins) +int workingset_report_intervals_parse(char *src, + struct wsr_report_bins *bins) { int err =3D 0, i =3D 0; char *cur, *next =3D strim(src); @@ -542,5 +545,8 @@ void notify_workingset(struct mem_cgroup *memcg, struct= pglist_data *pgdat) { struct wsr_state *wsr =3D &mem_cgroup_lruvec(memcg, pgdat)->wsr; =20 - kernfs_notify(wsr->page_age_sys_file); + if (mem_cgroup_is_root(memcg)) + kernfs_notify(wsr->page_age_sys_file); + else + cgroup_file_notify(wsr->page_age_cgroup_file); } --=20 2.45.1.467.gbab1589fc0-goog From nobody Fri Feb 13 01:40:26 2026 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A0EA613F451 for ; Tue, 4 Jun 2024 02:06:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717466781; cv=none; b=MAYkP4de7K/m6PQievwiCJkPr0BwMvOPwoyZMxibgsb+5YMY940yCAab3XiQyA5yzwqmRIaRi5wh00tqNm5YPCVvBd6vXZGnNs0o65fvnydbkQYCrXCx45kNt6eHGizEt1MT2bOvnFs89PiwHTvHhkH9EcHeGhR+A+CeP9lnU7o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717466781; c=relaxed/simple; bh=S8VvyqsFvFB13E45h+8KtTwj62d7eCeuVaa+khQBLAc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=h2cBk4rW8BJLYqzk2geEsh25se6ZtKG13zLiGs4zITA6DrSywopzYm1moTo9E1tI2oOq0aJ3nw6IMTIaJ0QryJTHkrbYQJ39veXBxVv0QF3rb9IlSyfA7dN4qNaSBYxBICiDtSIgRFH0kVKghXQWxJomU1FmsHu0lSWtzv7cQD0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--yuanchu.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=bUL100eE; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--yuanchu.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="bUL100eE" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dfa61b0b163so7972940276.3 for ; Mon, 03 Jun 2024 19:06:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717466778; x=1718071578; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IFb3qAFC4K5HEmXTa3qV09z8mTyiAg8MnrqMh3hMESY=; b=bUL100eEgCjxL5OqRvGoXBLg/BNNSrZYDbmQBpZMQHxt+Lj6PuR8RCGtcGGzfuWWsR 03pUkXIDxf27Vzoss6TEkcxNSDgZmerexSQ1E73oLhqMxZFFPM2/8tZNps5PxZi89C5U Ri227KwaiCN2BuZXWhnFVDyEWpZU1bZVBVaEjPdyZD82tkY4AAjphaTRUT2m4P3OFSYq hlbc26J5wxDnkFQGGph0DQXi3yUqUgGeDmh1I+6jGvEY7v08rO0Y9pZ2bWEvOid0Iua+ DjfPPAvfq7ckq69z1PAY29DtxzchWLOhgwFnQbYkqr1dHwG7czx/poPibhWqELSAcUaB XN4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717466778; x=1718071578; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IFb3qAFC4K5HEmXTa3qV09z8mTyiAg8MnrqMh3hMESY=; b=wToQSr45NqaCriFSexWRh+lLOHlVkqgKIhPiG0RL4PpaJhLSVyMOJbLbWKWNJgcxkb cJt+Dzi/ukHBlZUrc8z/vce4zvwDn7VqZS5VMTuuucVLXwAXavAuysNopch4gJHJIk/P T7eyyCCa+OyZXGM1PvDDcvEaNJJrStBUE7lKMticI5cYBnzQmUVUjzAnXa6rEq77rwYY K+I4ns2JmjjeOTxXwWRPR+wf6KSR/nH7abBSizx2zzYfutOqWnE868AM3yqSH6F+0E66 S7oJXifDP8kBjaIAHvPgu24cnt5HnDES9Nnm2a3yOIxnhRz/5FmIBEamaNPHQMfd6WSF yQiQ== X-Forwarded-Encrypted: i=1; AJvYcCWIIuQJZIJ8ZT1egzvLd+ssXrMrlGbz4uQ+pxywUZs9gwr9UMyItJ3jf4drqJ2Rqo3Cc07XedknLE3MlLaz4lXzit4uSxUUDCpSc9H7 X-Gm-Message-State: AOJu0YzqnaSnA2nyuaD985dDE28sa6Dv3fleRql+eFnA0vsIG7v+sYjh E4NgGdw11JiZ4YvI/ly6kZhTc0/Yj9n+XDvkxE5xcxud4+Qg2dTWSteXcLImu3eMb8toUDbg/X9 AD2Qb7A== X-Google-Smtp-Source: AGHT+IERBr3rpHyGYQbtWTU2Cnqq5B/QCSj2QzI+uA89IMrXbKRLeHdRMrnyxldq/MA/nBeGIemY82Zq3hyI X-Received: from yuanchu-desktop.svl.corp.google.com ([2620:15c:2a3:200:367f:7387:3dd2:73f1]) (user=yuanchu job=sendgmr) by 2002:a05:6902:706:b0:dfa:6ea5:c8d5 with SMTP id 3f1490d57ef6-dfa73d9b762mr2674608276.10.1717466778461; Mon, 03 Jun 2024 19:06:18 -0700 (PDT) Date: Mon, 3 Jun 2024 19:05:47 -0700 In-Reply-To: <20240604020549.1017540-1-yuanchu@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240604020549.1017540-1-yuanchu@google.com> X-Mailer: git-send-email 2.45.1.467.gbab1589fc0-goog Message-ID: <20240604020549.1017540-7-yuanchu@google.com> Subject: [PATCH v2 6/8] mm: add kernel aging thread for workingset reporting From: Yuanchu Xie To: David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Muhammad Usama Anjum Cc: Kalesh Singh , Wei Xu , David Rientjes , Greg Kroah-Hartman , "Rafael J. Wysocki" , Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Shuah Khan , Yosry Ahmed , Matthew Wilcox , Sudarshan Rajagopalan , Kairui Song , "Michael S. Tsirkin" , Vasily Averin , Nhat Pham , Miaohe Lin , Qi Zheng , Abel Wu , "Vishal Moola (Oracle)" , Kefeng Wang , Yuanchu Xie , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For reliable and timely aging on memcgs, one has to read the page age histograms on time. A kernel thread makes it easier by aging memcgs with valid refresh_interval when they can be refreshed, and also reduces the latency of any userspace consumers of the page age histogram. The kerne aging thread is gated behind CONFIG_WORKINGSET_REPORT_AGING. Debugging stats may be added in the future for when aging cannot keep up with the configured refresh_interval. Signed-off-by: Yuanchu Xie --- include/linux/workingset_report.h | 11 ++- mm/Kconfig | 6 ++ mm/Makefile | 1 + mm/memcontrol.c | 8 +- mm/workingset_report.c | 15 +++- mm/workingset_report_aging.c | 127 ++++++++++++++++++++++++++++++ 6 files changed, 162 insertions(+), 6 deletions(-) create mode 100644 mm/workingset_report_aging.c diff --git a/include/linux/workingset_report.h b/include/linux/workingset_r= eport.h index ae412d408037..9294023db5a8 100644 --- a/include/linux/workingset_report.h +++ b/include/linux/workingset_report.h @@ -63,7 +63,16 @@ void wsr_remove_sysfs(struct node *node); * The next refresh time is stored in refresh_time. */ bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, - struct pglist_data *pgdat); + struct pglist_data *pgdat, unsigned long *refresh_time); + +#ifdef CONFIG_WORKINGSET_REPORT_AGING +void wsr_wakeup_aging_thread(void); +#else /* CONFIG_WORKINGSET_REPORT_AGING */ +static inline void wsr_wakeup_aging_thread(void) +{ +} +#endif /* CONFIG_WORKINGSET_REPORT_AGING */ + #else static inline void wsr_init_lruvec(struct lruvec *lruvec) { diff --git a/mm/Kconfig b/mm/Kconfig index 03927ed2adbd..f8ff41408b9c 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1258,6 +1258,12 @@ config WORKINGSET_REPORT This option exports stats and events giving the user more insight into its memory working set. =20 +config WORKINGSET_REPORT_AGING + bool "Workingset report kernel aging thread" + depends on WORKINGSET_REPORT + help + Performs aging on memcgs with their configured refresh intervals. + source "mm/damon/Kconfig" =20 endmenu diff --git a/mm/Makefile b/mm/Makefile index ed05af2bb3e3..e9c9048e1e09 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -97,6 +97,7 @@ obj-$(CONFIG_TRANSPARENT_HUGEPAGE) +=3D huge_memory.o khu= gepaged.o obj-$(CONFIG_PAGE_COUNTER) +=3D page_counter.o obj-$(CONFIG_MEMCG) +=3D memcontrol.o vmpressure.o obj-$(CONFIG_WORKINGSET_REPORT) +=3D workingset_report.o +obj-$(CONFIG_WORKINGSET_REPORT_AGING) +=3D workingset_report_aging.o ifdef CONFIG_SWAP obj-$(CONFIG_MEMCG) +=3D swap_cgroup.o endif diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 48cdc9422794..547a2161b7e2 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -7281,12 +7281,12 @@ static ssize_t memory_ws_refresh_interval_write(str= uct kernfs_open_file *of, { unsigned int nid, msecs; struct wsr_state *wsr; + unsigned long old_interval; struct mem_cgroup *memcg =3D mem_cgroup_from_css(of_css(of)); ssize_t ret =3D memory_wsr_threshold_parse(buf, nbytes, &nid, &msecs); =20 if (ret < 0) return ret; - wsr =3D &mem_cgroup_lruvec(memcg, NODE_DATA(nid))->wsr; =20 mutex_lock(&wsr->page_age_lock); @@ -7305,9 +7305,13 @@ static ssize_t memory_ws_refresh_interval_write(stru= ct kernfs_open_file *of, wsr->page_age =3D NULL; } =20 + old_interval =3D READ_ONCE(wsr->refresh_interval); WRITE_ONCE(wsr->refresh_interval, msecs_to_jiffies(msecs)); unlock: mutex_unlock(&wsr->page_age_lock); + if (ret > 0 && msecs && + (!old_interval || jiffies_to_msecs(old_interval) > msecs)) + wsr_wakeup_aging_thread(); return ret; } =20 @@ -7358,7 +7362,7 @@ static int memory_ws_page_age_show(struct seq_file *m= , void *v) if (!READ_ONCE(wsr->page_age)) continue; =20 - wsr_refresh_report(wsr, memcg, NODE_DATA(nid)); + wsr_refresh_report(wsr, memcg, NODE_DATA(nid), NULL); mutex_lock(&wsr->page_age_lock); if (!wsr->page_age) goto unlock; diff --git a/mm/workingset_report.c b/mm/workingset_report.c index 72f2cad85a0d..15118c4aecb1 100644 --- a/mm/workingset_report.c +++ b/mm/workingset_report.c @@ -274,7 +274,7 @@ static void copy_node_bins(struct pglist_data *pgdat, } =20 bool wsr_refresh_report(struct wsr_state *wsr, struct mem_cgroup *root, - struct pglist_data *pgdat) + struct pglist_data *pgdat, unsigned long *refresh_time) { struct wsr_page_age_histo *page_age; unsigned long refresh_interval =3D READ_ONCE(wsr->refresh_interval); @@ -291,10 +291,14 @@ bool wsr_refresh_report(struct wsr_state *wsr, struct= mem_cgroup *root, goto unlock; if (page_age->timestamp && time_is_after_jiffies(page_age->timestamp + refresh_interval)) - goto unlock; + goto time; refresh_scan(wsr, root, pgdat, refresh_interval); copy_node_bins(pgdat, page_age); refresh_aggregate(page_age, root, pgdat); + +time: + if (refresh_time) + *refresh_time =3D page_age->timestamp + refresh_interval; unlock: mutex_unlock(&wsr->page_age_lock); return !!page_age; @@ -357,6 +361,7 @@ static ssize_t refresh_interval_store(struct kobject *k= obj, unsigned int interval; int err; struct wsr_state *wsr =3D kobj_to_wsr(kobj); + unsigned long old_interval =3D 0; =20 err =3D kstrtouint(buf, 0, &interval); if (err) @@ -378,9 +383,13 @@ static ssize_t refresh_interval_store(struct kobject *= kobj, wsr->page_age =3D NULL; } =20 + old_interval =3D READ_ONCE(wsr->refresh_interval); WRITE_ONCE(wsr->refresh_interval, msecs_to_jiffies(interval)); unlock: mutex_unlock(&wsr->page_age_lock); + if (!err && interval && + (!old_interval || jiffies_to_msecs(old_interval) > interval)) + wsr_wakeup_aging_thread(); return err ?: len; } =20 @@ -470,7 +479,7 @@ static ssize_t page_age_show(struct kobject *kobj, stru= ct kobj_attribute *attr, int ret =3D 0; struct wsr_state *wsr =3D kobj_to_wsr(kobj); =20 - wsr_refresh_report(wsr, NULL, kobj_to_pgdat(kobj)); + wsr_refresh_report(wsr, NULL, kobj_to_pgdat(kobj), NULL); =20 mutex_lock(&wsr->page_age_lock); if (!wsr->page_age) diff --git a/mm/workingset_report_aging.c b/mm/workingset_report_aging.c new file mode 100644 index 000000000000..91ad5020778a --- /dev/null +++ b/mm/workingset_report_aging.c @@ -0,0 +1,127 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Workingset report kernel aging thread + * + * Performs aging on behalf of memcgs with their configured refresh interv= al. + * While a userspace program can periodically read the page age breakdown + * per-memcg and trigger aging, the kernel performing aging is less overhe= ad, + * more consistent, and more reliable for the use case where every memcg s= hould + * be aged according to their refresh interval. + */ +#define pr_fmt(fmt) "workingset report aging: " fmt + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static DECLARE_WAIT_QUEUE_HEAD(aging_wait); +static bool refresh_pending; + +static bool do_aging_node(int nid, unsigned long *next_wake_time) +{ + struct mem_cgroup *memcg; + bool should_wait =3D true; + struct pglist_data *pgdat =3D NODE_DATA(nid); + + memcg =3D mem_cgroup_iter(NULL, NULL, NULL); + do { + struct lruvec *lruvec =3D mem_cgroup_lruvec(memcg, pgdat); + struct wsr_state *wsr =3D &lruvec->wsr; + unsigned long refresh_time; + + /* use returned time to decide when to wake up next */ + if (wsr_refresh_report(wsr, memcg, pgdat, &refresh_time)) { + if (should_wait) { + should_wait =3D false; + *next_wake_time =3D refresh_time; + } else if (time_before(refresh_time, *next_wake_time)) { + *next_wake_time =3D refresh_time; + } + } + + cond_resched(); + } while ((memcg =3D mem_cgroup_iter(NULL, memcg, NULL))); + + return should_wait; +} + +static int do_aging(void *unused) +{ + while (!kthread_should_stop()) { + int nid; + long timeout_ticks; + unsigned long next_wake_time; + bool should_wait =3D true; + + WRITE_ONCE(refresh_pending, false); + for_each_node_state(nid, N_MEMORY) { + unsigned long node_next_wake_time; + + if (do_aging_node(nid, &node_next_wake_time)) + continue; + if (should_wait) { + should_wait =3D false; + next_wake_time =3D node_next_wake_time; + } else if (time_before(node_next_wake_time, + next_wake_time)) { + next_wake_time =3D node_next_wake_time; + } + } + + if (should_wait) { + wait_event_interruptible(aging_wait, refresh_pending); + continue; + } + + /* sleep until next aging */ + timeout_ticks =3D next_wake_time - jiffies; + if (timeout_ticks > 0 && + timeout_ticks !=3D MAX_SCHEDULE_TIMEOUT) { + schedule_timeout_idle(timeout_ticks); + continue; + } + } + return 0; +} + +/* Invoked when refresh_interval shortens or changes to a non-zero value. = */ +void wsr_wakeup_aging_thread(void) +{ + WRITE_ONCE(refresh_pending, true); + wake_up_interruptible(&aging_wait); +} + +static struct task_struct *aging_thread; + +static int aging_init(void) +{ + struct task_struct *task; + + task =3D kthread_run(do_aging, NULL, "kagingd"); + + if (IS_ERR(task)) { + pr_err("Failed to create aging kthread\n"); + return PTR_ERR(task); + } + + aging_thread =3D task; + pr_info("module loaded\n"); + return 0; +} + +static void aging_exit(void) +{ + kthread_stop(aging_thread); + aging_thread =3D NULL; + pr_info("module unloaded\n"); +} + +module_init(aging_init); +module_exit(aging_exit); --=20 2.45.1.467.gbab1589fc0-goog From nobody Fri Feb 13 01:40:26 2026 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4D84A13FD86 for ; Tue, 4 Jun 2024 02:06:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717466783; cv=none; b=qug03qJ/+9dSqTGbYre1IGzcOhgUmxVW4E20bDLUIjqcetgRIOgOvyXa0qYD+zC/MiDe7sWK82JZ+c01gPiqlcZ4Nbcmn95M5MGNS4PIPHs030iPl4r/GPg9J6GWI2tx+YaYhorTlue6zMNWGMM9dDiWGSGVksGl8bmQ0OSELQI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717466783; c=relaxed/simple; bh=q3DyF7+oo54Jxd1cz3e3x7LUbFTHRR4P/k6/pzucz0o=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=S7RRiY+ZlKhC5hzdQOuoy4CxYnH64SLCI0EINtjnOzf5HNEMVOkkKsvItVNX+UsH1qblSbYO+lPeYIfapz032r2VFBQ5KbL4+8JTs5WueHR3iyGq03LdLEXAgW0VcPlttB4s6NYAXFC17mnmOrN8/+mbAPdys/dSBEP/eyNT6+A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--yuanchu.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=jV+VEndQ; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--yuanchu.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="jV+VEndQ" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-629f8a71413so86673067b3.1 for ; Mon, 03 Jun 2024 19:06:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717466780; x=1718071580; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GFXr85ron2toDBErygQNuv7Zt41ZsUWhKpGsReFw+QE=; b=jV+VEndQv8hB4kgqOTjuR6XQSlO4EKFKETM6l7XHJIpoNi0RCP5jOw9OD1IRLcq2NR g9EdUf+IRvPl5vQTIXNkDGgUU7uzNgZQnRgsW/BZagtLDP82zwq4nSmGLOBhF2kV5Rb6 RoXhOJhKcHTEiWxI9KugNINgn8D08wl19EWo1poPr4K5Jkya0i2TDtmNIxnx3G4B8WfQ TPGy7ITjiOW3mqJhZNg0lHSnnpyzukhNp+bzaL6kuxY+V83WvPNsIhjtGQ8ltrYWulzY 1YvuwKDIz1FwnyDaD3Yefy+JeJC8ORVnwma1OmaENpvTVl6KAutyXb+C6Fd9c6NhH0X+ LiFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717466780; x=1718071580; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GFXr85ron2toDBErygQNuv7Zt41ZsUWhKpGsReFw+QE=; b=GEvjh+lIuPnku1HyHYqQhBr61t0CGxU1IiTdED4/DUP9GPPA0NWf5pPRRlQIrnTqUs 4Kw5qfxuo7JFLFV0vsIx/8i+aUUrGp1twjEmxn9fbhvA4eGWP/9syqTJzCxmW9yAcECK QlvvYL5GpjLPcszi/CzWpMJD4QH76ol0Lie+/bRh9ZLtsQhg53g2Yk9hO4idq/0j4t1p xE+zLFYx77IZboZXkB9Q25SXpnHxls2GgYf4aarJL739Zw0nN2BzH/TqvWwnuLX9M3Im zyFgdLk77M1VJgKBvBYJav9VVv1TYnXBCl2AjcLbHmHL7EDdLfJnSyaD2mXLbob+CaGR P3Mg== X-Forwarded-Encrypted: i=1; AJvYcCVAM2rYBvc+hB/x+e9Ac+e8OhS+b9mEH8/J9wOXlQz/fTfxMtD8ld8Wq4DG6PouHIiMAU/PN4pqKtJzUBb3CFZ2HGPWAYGn45EfJZoK X-Gm-Message-State: AOJu0Yx/HfXftBCUKr+97e5yLluDyHuvYNs12hIH/Qt/5rHIrPvTrYu4 yHigc/tqxMAlTbEuaZzOLUm9B5J3sNOGzArYqyiiG6WcWodG8DmC+ktHgzUHBY9Zufi3RP2pl00 7PqK1HA== X-Google-Smtp-Source: AGHT+IFMlcay6XF4ua+TyOfXgCRZBAfcOugLyJT670B/gtTfpnXAqB6fBBX7EL4l62Xcg856n16+j8Gxc2Th X-Received: from yuanchu-desktop.svl.corp.google.com ([2620:15c:2a3:200:367f:7387:3dd2:73f1]) (user=yuanchu job=sendgmr) by 2002:a05:690c:a82:b0:627:d549:c40b with SMTP id 00721157ae682-62c7981f58amr32392627b3.5.1717466780405; Mon, 03 Jun 2024 19:06:20 -0700 (PDT) Date: Mon, 3 Jun 2024 19:05:48 -0700 In-Reply-To: <20240604020549.1017540-1-yuanchu@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240604020549.1017540-1-yuanchu@google.com> X-Mailer: git-send-email 2.45.1.467.gbab1589fc0-goog Message-ID: <20240604020549.1017540-8-yuanchu@google.com> Subject: [PATCH v2 7/8] selftest: test system-wide workingset reporting From: Yuanchu Xie To: David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Muhammad Usama Anjum Cc: Kalesh Singh , Wei Xu , David Rientjes , Greg Kroah-Hartman , "Rafael J. Wysocki" , Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Shuah Khan , Yosry Ahmed , Matthew Wilcox , Sudarshan Rajagopalan , Kairui Song , "Michael S. Tsirkin" , Vasily Averin , Nhat Pham , Miaohe Lin , Qi Zheng , Abel Wu , "Vishal Moola (Oracle)" , Kefeng Wang , Yuanchu Xie , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A basic test that verifies the working set size of a simple memory accessor. It should work with or without the aging thread. When running tests with run_vmtests.sh, file workingset report testing requires an environment variable WORKINGSET_REPORT_TEST_FILE_PATH to store a temporary file, which is passed into the test invocation as a parameter. Signed-off-by: Yuanchu Xie --- tools/testing/selftests/mm/.gitignore | 1 + tools/testing/selftests/mm/Makefile | 3 + tools/testing/selftests/mm/run_vmtests.sh | 5 + .../testing/selftests/mm/workingset_report.c | 306 ++++++++++++++++ .../testing/selftests/mm/workingset_report.h | 39 +++ .../selftests/mm/workingset_report_test.c | 329 ++++++++++++++++++ 6 files changed, 683 insertions(+) create mode 100644 tools/testing/selftests/mm/workingset_report.c create mode 100644 tools/testing/selftests/mm/workingset_report.h create mode 100644 tools/testing/selftests/mm/workingset_report_test.c diff --git a/tools/testing/selftests/mm/.gitignore b/tools/testing/selftest= s/mm/.gitignore index 0b9ab987601c..f1570923019d 100644 --- a/tools/testing/selftests/mm/.gitignore +++ b/tools/testing/selftests/mm/.gitignore @@ -49,3 +49,4 @@ hugetlb_fault_after_madv hugetlb_madv_vs_map mseal_test seal_elf +workingset_report_test diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/= mm/Makefile index 3b49bc3d0a3b..6c96a65078f2 100644 --- a/tools/testing/selftests/mm/Makefile +++ b/tools/testing/selftests/mm/Makefile @@ -73,6 +73,7 @@ TEST_GEN_FILES +=3D ksm_functional_tests TEST_GEN_FILES +=3D mdwe_test TEST_GEN_FILES +=3D hugetlb_fault_after_madv TEST_GEN_FILES +=3D hugetlb_madv_vs_map +TEST_GEN_FILES +=3D workingset_report_test =20 ifneq ($(ARCH),arm64) TEST_GEN_FILES +=3D soft-dirty @@ -131,6 +132,8 @@ $(TEST_GEN_FILES): vm_util.c thp_settings.c $(OUTPUT)/uffd-stress: uffd-common.c $(OUTPUT)/uffd-unit-tests: uffd-common.c =20 +$(OUTPUT)/workingset_report_test: workingset_report.c + ifeq ($(ARCH),x86_64) BINARIES_32 :=3D $(patsubst %,$(OUTPUT)/%,$(BINARIES_32)) BINARIES_64 :=3D $(patsubst %,$(OUTPUT)/%,$(BINARIES_64)) diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/self= tests/mm/run_vmtests.sh index 3157204b9047..41419084b481 100755 --- a/tools/testing/selftests/mm/run_vmtests.sh +++ b/tools/testing/selftests/mm/run_vmtests.sh @@ -75,6 +75,8 @@ separated by spaces: read-only VMAs - mdwe test prctl(PR_SET_MDWE, ...) +- workingset_report + test workingset reporting =20 example: ./run_vmtests.sh -t "hmm mmap ksm" EOF @@ -446,6 +448,9 @@ CATEGORY=3D"mkdirty" run_test ./mkdirty =20 CATEGORY=3D"mdwe" run_test ./mdwe_test =20 +CATEGORY=3D"workingset_report" run_test ./workingset_report_test \ + "${WORKINGSET_REPORT_TEST_FILE_PATH}" + echo "SUMMARY: PASS=3D${count_pass} SKIP=3D${count_skip} FAIL=3D${count_fa= il}" | tap_prefix echo "1..${count_total}" | tap_output =20 diff --git a/tools/testing/selftests/mm/workingset_report.c b/tools/testing= /selftests/mm/workingset_report.c new file mode 100644 index 000000000000..ee4dda5c371d --- /dev/null +++ b/tools/testing/selftests/mm/workingset_report.c @@ -0,0 +1,306 @@ +// SPDX-License-Identifier: GPL-2.0 +#include "workingset_report.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../kselftest.h" + +#define SYSFS_NODE_ONLINE "/sys/devices/system/node/online" +#define PROC_DROP_CACHES "/proc/sys/vm/drop_caches" + +/* Returns read len on success, or -errno on failure. */ +static ssize_t read_text(const char *path, char *buf, size_t max_len) +{ + ssize_t len; + int fd, err; + size_t bytes_read =3D 0; + + if (!max_len) + return -EINVAL; + + fd =3D open(path, O_RDONLY); + if (fd < 0) + return -errno; + + while (bytes_read < max_len - 1) { + len =3D read(fd, buf + bytes_read, max_len - 1 - bytes_read); + + if (len <=3D 0) + break; + bytes_read +=3D len; + } + + buf[bytes_read] =3D '\0'; + + err =3D -errno; + close(fd); + return len < 0 ? err : bytes_read; +} + +/* Returns written len on success, or -errno on failure. */ +static ssize_t write_text(const char *path, const char *buf, ssize_t max_l= en) +{ + int fd, len, err; + size_t bytes_written =3D 0; + + fd =3D open(path, O_WRONLY | O_APPEND); + if (fd < 0) + return -errno; + + while (bytes_written < max_len) { + len =3D write(fd, buf + bytes_written, max_len - bytes_written); + + if (len < 0) + break; + bytes_written +=3D len; + } + + err =3D -errno; + close(fd); + return len < 0 ? err : bytes_written; +} + +static long read_num(const char *path) +{ + char buf[21]; + + if (read_text(path, buf, sizeof(buf)) <=3D 0) + return -1; + return (long)strtoul(buf, NULL, 10); +} + +static int write_num(const char *path, unsigned long n) +{ + char buf[21]; + + sprintf(buf, "%lu", n); + if (write_text(path, buf, strlen(buf)) < 0) + return -1; + return 0; +} + +long sysfs_get_refresh_interval(int nid) +{ + char file[128]; + + snprintf(file, sizeof(file), + "/sys/devices/system/node/node%d/workingset_report/refresh_interval", + nid); + return read_num(file); +} + +int sysfs_set_refresh_interval(int nid, long interval) +{ + char file[128]; + + snprintf(file, sizeof(file), + "/sys/devices/system/node/node%d/workingset_report/refresh_interval", + nid); + return write_num(file, interval); +} + +int sysfs_get_page_age_intervals_str(int nid, char *buf, int len) +{ + char path[128]; + + snprintf(path, sizeof(path), + "/sys/devices/system/node/node%d/workingset_report/page_age_intervals", + nid); + return read_text(path, buf, len); + +} + +int sysfs_set_page_age_intervals_str(int nid, const char *buf, int len) +{ + char path[128]; + + snprintf(path, sizeof(path), + "/sys/devices/system/node/node%d/workingset_report/page_age_intervals", + nid); + return write_text(path, buf, len); +} + +int sysfs_set_page_age_intervals(int nid, const char *const intervals[], + int nr_intervals) +{ + char file[128]; + char buf[1024]; + int i; + int err, len =3D 0; + + for (i =3D 0; i < nr_intervals; ++i) { + err =3D snprintf(buf + len, sizeof(buf) - len, "%s", intervals[i]); + + if (err < 0) + return err; + len +=3D err; + + if (i < nr_intervals - 1) { + err =3D snprintf(buf + len, sizeof(buf) - len, ","); + if (err < 0) + return err; + len +=3D err; + } + } + + snprintf(file, sizeof(file), + "/sys/devices/system/node/node%d/workingset_report/page_age_intervals", + nid); + return write_text(file, buf, len); +} + +int get_nr_nodes(void) +{ + char buf[22]; + char *found; + + if (read_text(SYSFS_NODE_ONLINE, buf, sizeof(buf)) <=3D 0) + return -1; + found =3D strstr(buf, "-"); + if (found) + return (int)strtoul(found + 1, NULL, 10) + 1; + return (long)strtoul(buf, NULL, 10) + 1; +} + +int drop_pagecache(void) +{ + return write_num(PROC_DROP_CACHES, 1); +} + +ssize_t sysfs_page_age_read(int nid, char *buf, size_t len) + +{ + char file[128]; + + snprintf(file, sizeof(file), + "/sys/devices/system/node/node%d/workingset_report/page_age", + nid); + return read_text(file, buf, len); +} + +/* + * Finds the first occurrence of "N\n" + * Modifies buf to terminate before the next occurrence of "N". + * Returns a substring of buf starting after "N\n" + */ +char *page_age_split_node(char *buf, int nid, char **next) +{ + char node_str[5]; + char *found; + int node_str_len; + + node_str_len =3D snprintf(node_str, sizeof(node_str), "N%u\n", nid); + + /* find the node prefix first */ + found =3D strstr(buf, node_str); + if (!found) { + ksft_print_msg("cannot find '%s' in page_idle_age", node_str); + return NULL; + } + found +=3D node_str_len; + + *next =3D strchr(found, 'N'); + if (*next) + *(*next - 1) =3D '\0'; + + return found; +} + +ssize_t page_age_read(const char *buf, const char *interval, int pagetype) +{ + static const char * const type[ANON_AND_FILE] =3D { "anon=3D", "file=3D" = }; + char *found; + + found =3D strstr(buf, interval); + if (!found) { + ksft_print_msg("cannot find %s in page_age", interval); + return -1; + } + found =3D strstr(found, type[pagetype]); + if (!found) { + ksft_print_msg("cannot find %s in page_age", type[pagetype]); + return -1; + } + found +=3D strlen(type[pagetype]); + return (long)strtoul(found, NULL, 10); +} + +static const char *TEMP_FILE =3D "/tmp/workingset_selftest"; +void cleanup_file_workingset(void) +{ + remove(TEMP_FILE); +} + +int alloc_file_workingset(void *arg) +{ + int err =3D 0; + char *ptr; + int fd; + int ppid; + char *mapped; + size_t size =3D (size_t)arg; + size_t page_size =3D getpagesize(); + + ppid =3D getppid(); + + fd =3D open(TEMP_FILE, O_RDWR | O_CREAT); + if (fd < 0) { + err =3D -errno; + ksft_perror("failed to open temp file\n"); + goto cleanup; + } + + if (fallocate(fd, 0, 0, size) < 0) { + err =3D -errno; + ksft_perror("fallocate"); + goto cleanup; + } + + mapped =3D (char *)mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, + fd, 0); + if (mapped =3D=3D NULL) { + err =3D -errno; + ksft_perror("mmap"); + goto cleanup; + } + + while (getppid() =3D=3D ppid) { + sync(); + for (ptr =3D mapped; ptr < mapped + size; ptr +=3D page_size) + *ptr =3D *ptr ^ 0xFF; + } + +cleanup: + cleanup_file_workingset(); + return err; +} + +int alloc_anon_workingset(void *arg) +{ + char *buf, *ptr; + int ppid =3D getppid(); + size_t size =3D (size_t)arg; + size_t page_size =3D getpagesize(); + + buf =3D malloc(size); + + if (!buf) { + ksft_print_msg("cannot allocate anon workingset"); + exit(1); + } + + while (getppid() =3D=3D ppid) { + for (ptr =3D buf; ptr < buf + size; ptr +=3D page_size) + *ptr =3D *ptr ^ 0xFF; + } + + free(buf); + return 0; +} diff --git a/tools/testing/selftests/mm/workingset_report.h b/tools/testing= /selftests/mm/workingset_report.h new file mode 100644 index 000000000000..c5c281e4069b --- /dev/null +++ b/tools/testing/selftests/mm/workingset_report.h @@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef WORKINGSET_REPORT_H_ +#define WORKINGSET_REPORT_H_ + +#ifndef _GNU_SOURCE +#define _GNU_SOURCE +#endif + +#include +#include +#include +#include +#include + +#define PAGETYPE_ANON 0 +#define PAGETYPE_FILE 1 +#define ANON_AND_FILE 2 + +int get_nr_nodes(void); +int drop_pagecache(void); + +long sysfs_get_refresh_interval(int nid); +int sysfs_set_refresh_interval(int nid, long interval); + +int sysfs_get_page_age_intervals_str(int nid, char *buf, int len); +int sysfs_set_page_age_intervals_str(int nid, const char *buf, int len); + +int sysfs_set_page_age_intervals(int nid, const char *const intervals[], + int nr_intervals); + +char *page_age_split_node(char *buf, int nid, char **next); +ssize_t sysfs_page_age_read(int nid, char *buf, size_t len); +ssize_t page_age_read(const char *buf, const char *interval, int pagetype); + +int alloc_file_workingset(void *arg); +void cleanup_file_workingset(void); +int alloc_anon_workingset(void *arg); + +#endif /* WORKINGSET_REPORT_H_ */ diff --git a/tools/testing/selftests/mm/workingset_report_test.c b/tools/te= sting/selftests/mm/workingset_report_test.c new file mode 100644 index 000000000000..73246f18ed8d --- /dev/null +++ b/tools/testing/selftests/mm/workingset_report_test.c @@ -0,0 +1,329 @@ +// SPDX-License-Identifier: GPL-2.0 +#include "workingset_report.h" + +#include +#include +#include +#include + +#include "../clone3/clone3_selftests.h" + +#define REFRESH_INTERVAL 5000 +#define MB(x) (x << 20) + +static void sleep_ms(int milliseconds) +{ + struct timespec ts; + + ts.tv_sec =3D milliseconds / 1000; + ts.tv_nsec =3D (milliseconds % 1000) * 1000000; + nanosleep(&ts, NULL); +} + +/* + * Checks if two given values differ by less than err% of their sum. + */ +static inline int values_close(long a, long b, int err) +{ + return labs(a - b) <=3D (a + b) / 100 * err; +} + +static const char * const PAGE_AGE_INTERVALS[] =3D { + "6000", "10000", "15000", "18446744073709551615", +}; +#define NR_PAGE_AGE_INTERVALS (ARRAY_SIZE(PAGE_AGE_INTERVALS)) + +static int set_page_age_intervals_all_nodes(const char *intervals, int nr_= nodes) +{ + int i; + + for (i =3D 0; i < nr_nodes; ++i) { + int err =3D sysfs_set_page_age_intervals_str( + i, &intervals[i * 1024], strlen(&intervals[i * 1024])); + + if (err < 0) + return err; + } + return 0; +} + +static int get_page_age_intervals_all_nodes(char *intervals, int nr_nodes) +{ + int i; + + for (i =3D 0; i < nr_nodes; ++i) { + int err =3D sysfs_get_page_age_intervals_str( + i, &intervals[i * 1024], 1024); + + if (err < 0) + return err; + } + return 0; +} + +static int set_refresh_interval_all_nodes(const long *interval, int nr_nod= es) +{ + int i; + + for (i =3D 0; i < nr_nodes; ++i) { + int err =3D sysfs_set_refresh_interval(i, interval[i]); + + if (err < 0) + return err; + } + return 0; +} + +static int get_refresh_interval_all_nodes(long *interval, int nr_nodes) +{ + int i; + + for (i =3D 0; i < nr_nodes; ++i) { + long val =3D sysfs_get_refresh_interval(i); + + if (val < 0) + return val; + interval[i] =3D val; + } + return 0; +} + +static pid_t clone_and_run(int fn(void *arg), void *arg) +{ + pid_t pid; + + struct __clone_args args =3D { + .exit_signal =3D SIGCHLD, + }; + + pid =3D sys_clone3(&args, sizeof(struct __clone_args)); + + if (pid =3D=3D 0) + exit(fn(arg)); + + return pid; +} + +static int read_workingset(int pagetype, int nid, + unsigned long page_age[NR_PAGE_AGE_INTERVALS]) +{ + int i, err; + char buf[4096]; + + err =3D sysfs_page_age_read(nid, buf, sizeof(buf)); + if (err < 0) + return err; + + for (i =3D 0; i < NR_PAGE_AGE_INTERVALS; ++i) { + err =3D page_age_read(buf, PAGE_AGE_INTERVALS[i], pagetype); + if (err < 0) + return err; + page_age[i] =3D err; + } + + return 0; +} + +static ssize_t read_interval_all_nodes(int pagetype, int interval) +{ + int i, err; + unsigned long page_age[NR_PAGE_AGE_INTERVALS]; + ssize_t ret =3D 0; + int nr_nodes =3D get_nr_nodes(); + + for (i =3D 0; i < nr_nodes; ++i) { + err =3D read_workingset(pagetype, i, page_age); + if (err < 0) + return err; + + ret +=3D page_age[interval]; + } + + return ret; +} + +#define TEST_SIZE MB(500l) + +static int run_test(int f(void)) +{ + int i, err, test_result; + long *old_refresh_intervals; + long *new_refresh_intervals; + char *old_page_age_intervals; + int nr_nodes =3D get_nr_nodes(); + + if (nr_nodes <=3D 0) { + ksft_print_msg("failed to get nr_nodes\n"); + return KSFT_FAIL; + } + + old_refresh_intervals =3D calloc(nr_nodes, sizeof(long)); + new_refresh_intervals =3D calloc(nr_nodes, sizeof(long)); + old_page_age_intervals =3D calloc(nr_nodes, 1024); + + if (!(old_refresh_intervals && new_refresh_intervals && + old_page_age_intervals)) { + ksft_print_msg("failed to allocate memory for intervals\n"); + return KSFT_FAIL; + } + + err =3D get_refresh_interval_all_nodes(old_refresh_intervals, nr_nodes); + if (err < 0) { + ksft_print_msg("failed to read refresh interval\n"); + return KSFT_FAIL; + } + + err =3D get_page_age_intervals_all_nodes(old_page_age_intervals, nr_nodes= ); + if (err < 0) { + ksft_print_msg("failed to read page age interval\n"); + return KSFT_FAIL; + } + + for (i =3D 0; i < nr_nodes; ++i) + new_refresh_intervals[i] =3D REFRESH_INTERVAL; + + for (i =3D 0; i < nr_nodes; ++i) { + err =3D sysfs_set_page_age_intervals(i, PAGE_AGE_INTERVALS, + NR_PAGE_AGE_INTERVALS - 1); + if (err < 0) { + ksft_print_msg("failed to set page age interval\n"); + test_result =3D KSFT_FAIL; + goto fail; + } + } + + err =3D set_refresh_interval_all_nodes(new_refresh_intervals, nr_nodes); + if (err < 0) { + ksft_print_msg("failed to set refresh interval\n"); + test_result =3D KSFT_FAIL; + goto fail; + } + + sync(); + drop_pagecache(); + + test_result =3D f(); + +fail: + err =3D set_refresh_interval_all_nodes(old_refresh_intervals, nr_nodes); + if (err < 0) { + ksft_print_msg("failed to restore refresh interval\n"); + test_result =3D KSFT_FAIL; + } + err =3D set_page_age_intervals_all_nodes(old_page_age_intervals, nr_nodes= ); + if (err < 0) { + ksft_print_msg("failed to restore page age interval\n"); + test_result =3D KSFT_FAIL; + } + return test_result; +} + +static char *file_test_path; +static int test_file(void) +{ + ssize_t ws_size_ref, ws_size_test; + int ret =3D KSFT_FAIL, i; + pid_t pid =3D 0; + + if (!file_test_path) { + ksft_print_msg("Set a path to test file workingset\n"); + return KSFT_SKIP; + } + + ws_size_ref =3D read_interval_all_nodes(PAGETYPE_FILE, 0); + if (ws_size_ref < 0) + goto cleanup; + + pid =3D clone_and_run(alloc_file_workingset, (void *)TEST_SIZE); + if (pid < 0) + goto cleanup; + + read_interval_all_nodes(PAGETYPE_FILE, 0); + sleep_ms(REFRESH_INTERVAL); + + for (i =3D 0; i < 3; ++i) { + sleep_ms(REFRESH_INTERVAL); + ws_size_test =3D read_interval_all_nodes(PAGETYPE_FILE, 0); + ws_size_test +=3D read_interval_all_nodes(PAGETYPE_FILE, 1); + if (ws_size_test < 0) + goto cleanup; + + if (!values_close(ws_size_test - ws_size_ref, TEST_SIZE, 10)) { + ksft_print_msg( + "file working set size difference too large: actual=3D%ld, expected=3D= %ld\n", + ws_size_test - ws_size_ref, TEST_SIZE); + goto cleanup; + } + } + ret =3D KSFT_PASS; + +cleanup: + if (pid > 0) + kill(pid, SIGKILL); + cleanup_file_workingset(); + return ret; +} + +static int test_anon(void) +{ + ssize_t ws_size_ref, ws_size_test; + pid_t pid =3D 0; + int ret =3D KSFT_FAIL, i; + + ws_size_ref =3D read_interval_all_nodes(PAGETYPE_ANON, 0); + if (ws_size_ref < 0) + goto cleanup; + + pid =3D clone_and_run(alloc_anon_workingset, (void *)TEST_SIZE); + if (pid < 0) + goto cleanup; + + sleep_ms(REFRESH_INTERVAL); + read_interval_all_nodes(PAGETYPE_ANON, 0); + + for (i =3D 0; i < 5; ++i) { + sleep_ms(REFRESH_INTERVAL); + ws_size_test =3D read_interval_all_nodes(PAGETYPE_ANON, 0); + ws_size_test +=3D read_interval_all_nodes(PAGETYPE_ANON, 1); + if (ws_size_test < 0) + goto cleanup; + + if (!values_close(ws_size_test - ws_size_ref, TEST_SIZE, 10)) { + ksft_print_msg( + "anon working set size difference too large: actual=3D%ld, expected=3D= %ld\n", + ws_size_test - ws_size_ref, TEST_SIZE); + goto cleanup; + } + } + ret =3D KSFT_PASS; + +cleanup: + if (pid > 0) + kill(pid, SIGKILL); + return ret; +} + + +#define T(x) { x, #x } +struct workingset_test { + int (*fn)(void); + const char *name; +} tests[] =3D { + T(test_anon), + T(test_file), +}; +#undef T + +int main(int argc, char **argv) +{ + int i, err; + + if (argc > 1) + file_test_path =3D argv[1]; + + for (i =3D 0; i < ARRAY_SIZE(tests); i++) { + err =3D run_test(tests[i].fn); + ksft_test_result_code(err, tests[i].name, NULL); + } + return 0; +} --=20 2.45.1.467.gbab1589fc0-goog From nobody Fri Feb 13 01:40:26 2026 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5966514039E for ; Tue, 4 Jun 2024 02:06:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717466785; cv=none; b=Ue7yjlzV5AJnCZ84jYbKjnKU4uRwv7JYWC8l2+P9w4P1ucgWi2x2GCWYaOwmPIx+PreshWe2YPr98S2ftXuE9ECVfSQ3tUrEoHLq3Kb2cIGpMJC6XXYhLMGxGW0A4+xGi0J57bCTNCONbQChaF3YQFPkZDhqkZ+siei/4P32mIs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717466785; c=relaxed/simple; bh=eASpyk4WWK4m1u5bAgzdZmzHoLxlNRCkiIkqJ+AFaTw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=hD4+3TfXv9L+9xTE5xmqu7fBD7Cb4KUgjZ+kTR6tZMoxRjt4iiWJZ5epDEvIWCoSTR4JhKrE1QYLPE54b/7y+WLJE4g4QEbe1BqagXZYCjF64SCVAvAt7qJzLDQvhoubHGs80hHPm+b9lxGPYhccNkBNG89G6qJCpMd1ZSWijWI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--yuanchu.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=kw3CUsG4; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--yuanchu.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="kw3CUsG4" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-62a3dec382eso54944237b3.1 for ; Mon, 03 Jun 2024 19:06:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717466782; x=1718071582; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TnO0wJpr9VrNuJfB7YNxreIL5h7wQch0G6mtECTuPNI=; b=kw3CUsG4HuXRbBfwPxIlmqbR59+zdoruCRpZfL487nlZ/KeYrxAz5IYal2lceH7KcQ M090USlbcJr0vsXC4S19u/7Wl64ULN/7p+VrkpqJB2xukoqq/ZTtsK2BND/MKg5tcRP4 g68Zq1Lyl0xQ8Qyos4iFJUsv2uRnYVQV77jpfMKFYy8gHKVKh+k99f9oDe9eijg5ov1r GiveBNjP+eolSf94p441u8j3HDuZO8hjKV63kEie0Og9IeVVt7JVw1FbSj+Luv6/vNz4 49+OPKcDmrWbAf9Nn7Mm+iKp8S/jQisMIqWGJekmOVrt92iCG9X1IkSqj0yC5GpD1vVM 1Sxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717466782; x=1718071582; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TnO0wJpr9VrNuJfB7YNxreIL5h7wQch0G6mtECTuPNI=; b=LvP0oABTM6fu3/ziucsiVwEXQzftFGHZpSNfIVa7KCgKCQb3TELG/a38uYtphAuMyi 7YlcFfVY7UN2nYQIMfcnt6++EwAVHxUkuwOtsxci3Kr63iVAL5uHfcBkx65Smb640mBX P6uwLJxzBtEmcBXq1UKe+ZfmpFPUiWTtvRnMCYzH44yNnDTVOxfhlyA1ISzzjAXL99jV cnvUJB3tdZRBywNFSKV+X1+moKOQp7B9atqWNidw7rw64jToKozS8uXovVsaVrP2fyfM tDU4/jPVwNFMBwVPdRQxiGqD/+tkxRZnkDb4+dpZRJenvkJQv/diIs5pcfUjMUdz/Tee 8d4w== X-Forwarded-Encrypted: i=1; AJvYcCW2z9MyiZFGTRK91XLHc0jxxp70P6Dj7pd1UWiaDUhIKVkltivytpBdotB99d5EWZpNlGBIwi35QtUDQg7+8gLKiXrjOsmdXCri5bCj X-Gm-Message-State: AOJu0YxgsOmCC6sv+hpXl89FvROoekrZlXUNeLm1Cn/54SUVaQmqxISx BkxKYoMhWvw3NC3bL6SA7qopMU6tNhg3Bk1NWeKzhEFL5peb7+T0v/UjCIDghL29EsdNvohN4bJ UlhrI8Q== X-Google-Smtp-Source: AGHT+IGZTKhErP0mteEdLmLII+zOO1bIGHOtdpRtyYGuauHmWRG/AEKax0/K3QpTHO4KkINtPULvAXTKU6Gn X-Received: from yuanchu-desktop.svl.corp.google.com ([2620:15c:2a3:200:367f:7387:3dd2:73f1]) (user=yuanchu job=sendgmr) by 2002:a81:c906:0:b0:618:9348:6b92 with SMTP id 00721157ae682-62cabc4cd34mr3112857b3.1.1717466782241; Mon, 03 Jun 2024 19:06:22 -0700 (PDT) Date: Mon, 3 Jun 2024 19:05:49 -0700 In-Reply-To: <20240604020549.1017540-1-yuanchu@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240604020549.1017540-1-yuanchu@google.com> X-Mailer: git-send-email 2.45.1.467.gbab1589fc0-goog Message-ID: <20240604020549.1017540-9-yuanchu@google.com> Subject: [PATCH v2 8/8] Docs/admin-guide/mm/workingset_report: document sysfs and memcg interfaces From: Yuanchu Xie To: David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Huang Ying , Muhammad Usama Anjum Cc: Kalesh Singh , Wei Xu , David Rientjes , Greg Kroah-Hartman , "Rafael J. Wysocki" , Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Shuah Khan , Yosry Ahmed , Matthew Wilcox , Sudarshan Rajagopalan , Kairui Song , "Michael S. Tsirkin" , Vasily Averin , Nhat Pham , Miaohe Lin , Qi Zheng , Abel Wu , "Vishal Moola (Oracle)" , Kefeng Wang , Yuanchu Xie , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add workingset reporting documentation for better discoverability of its sysfs and memcg interfaces. Also document the required kernel config to enable workingset reporting. Signed-off-by: Yuanchu Xie --- Documentation/admin-guide/mm/index.rst | 1 + .../admin-guide/mm/workingset_report.rst | 105 ++++++++++++++++++ 2 files changed, 106 insertions(+) create mode 100644 Documentation/admin-guide/mm/workingset_report.rst diff --git a/Documentation/admin-guide/mm/index.rst b/Documentation/admin-g= uide/mm/index.rst index 1f883abf3f00..fba987de8997 100644 --- a/Documentation/admin-guide/mm/index.rst +++ b/Documentation/admin-guide/mm/index.rst @@ -41,4 +41,5 @@ the Linux memory management. swap_numa transhuge userfaultfd + workingset_report zswap diff --git a/Documentation/admin-guide/mm/workingset_report.rst b/Documenta= tion/admin-guide/mm/workingset_report.rst new file mode 100644 index 000000000000..f455ae93b30e --- /dev/null +++ b/Documentation/admin-guide/mm/workingset_report.rst @@ -0,0 +1,105 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Workingset Report +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Workingset report provides a view of memory coldness in user-defined +time intervals, i.e. X bytes are Y milliseconds cold. It breaks down +the user pages in the system per-NUMA node, per-memcg, for both +anonymous and file pages into histograms that look like: +:: + + 1000 anon=3D137368 file=3D24530 + 20000 anon=3D34342 file=3D0 + 30000 anon=3D353232 file=3D333608 + 40000 anon=3D407198 file=3D206052 + 9223372036854775807 anon=3D4925624 file=3D892892 + +The workingset reports can be used to drive proactive reclaim, by +identifying the number of cold bytes in a memcg, then writing to +``memory.reclaim``. + +Quick start +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Build the kernel with the following configurations. The report relies +on Multi-gen LRU for page coldness. + +* ``CONFIG_LRU_GEN=3Dy`` +* ``CONFIG_LRU_GEN_ENABLED=3Dy`` +* ``CONFIG_WORKINGSET_REPORT=3Dy`` + +Optionally, the aging kernel daemon can be enabled with the following +configuration. +* ``CONFIG_LRU_GEN_ENABLED=3Dy`` + +Sysfs interfaces +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +``/sys/devices/system/node/nodeX/page_age`` provides a per-node page +age histogram, showing an aggregate of the node's lruvecs. +Reading this file causes a hierarchical aging of all lruvecs, scanning +pages and creates a new Multi-gen LRU generation in each lruvec. +For example: +:: + + 1000 anon=3D0 file=3D0 + 2000 anon=3D0 file=3D0 + 100000 anon=3D5533696 file=3D5566464 + 18446744073709551615 anon=3D0 file=3D0 + +``/sys/devices/system/node/nodeX/page_age_interval`` is a comma +separated list of time in milliseconds that configures what the page +age histogram uses for aggregation. For the above histogram, +the intervals are: +:: + 1000,2000,100000 + +``/sys/devices/system/node/nodeX/workingset_report/refresh_interval`` +defines the amount of time the report is valid for in milliseconds. +When a report is still valid, reading the ``page_age`` file shows +the existing valid report, instead of generating a new one. + +``/sys/devices/system/node/nodeX/workingset_report/report_threshold`` +specifies how often the userspace agent can be notified for node +memory pressure, in milliseconds. When a node reaches its low +watermarks and wakes up kswapd, programs waiting on ``page_age`` are +woken up so they can read the histogram and make policy decisions. + +Memcg interface +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +While ``page_age_interval`` is defined per-node in sysfs. ``page_age``, +``refresh_interval`` and ``report_threshold`` are available per-memcg. + +``/sys/fs/cgroup/.../memory.workingset.page_age`` +The memcg equivalent of the sysfs workingset page age histogram, +breaks down the workingset of this memcg and its children into +page age intervals. Each node is prefixed with a node header and +a newline. Non-proactive direct reclaim on this memcg can also +wake up userspace agents that are waiting on this file. +e.g. +:: + + N0 + 1000 anon=3D0 file=3D0 + 2000 anon=3D0 file=3D0 + 3000 anon=3D0 file=3D0 + 4000 anon=3D0 file=3D0 + 5000 anon=3D0 file=3D0 + 18446744073709551615 anon=3D0 file=3D0 + +``/sys/fs/cgroup/.../memory.workingset.refresh_interval`` +The memcg equivalent of the sysfs refresh interval. A per-node +number of how much time a page age histogram is valid for, in +milliseconds. +e.g. +:: + + echo N0=3D2000 > memory.workingset.refresh_interval + +``/sys/fs/cgroup/.../memory.workingset.report_threshold`` +The memcg equivalent of the sysfs report threshold. A per-node +number of how often userspace agent waiting on the page age +histogram can be woken up, in milliseconds. +e.g. +:: + + echo N0=3D1000 > memory.workingset.report_threshold --=20 2.45.1.467.gbab1589fc0-goog