From nobody Sun Jun 28 01:48:40 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60B6EC433EF for ; Wed, 16 Feb 2022 08:30:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230126AbiBPIbD (ORCPT ); Wed, 16 Feb 2022 03:31:03 -0500 Received: from gmail-smtp-in.l.google.com ([23.128.96.19]:34846 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229510AbiBPIa7 (ORCPT ); Wed, 16 Feb 2022 03:30:59 -0500 Received: from out30-42.freemail.mail.aliyun.com (out30-42.freemail.mail.aliyun.com [115.124.30.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC9AA22BF7 for ; Wed, 16 Feb 2022 00:30:47 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R151e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04407;MF=xhao@linux.alibaba.com;NM=1;PH=DS;RN=6;SR=0;TI=SMTPD_---0V4cXjyS_1645000244; Received: from localhost.localdomain(mailfrom:xhao@linux.alibaba.com fp:SMTPD_---0V4cXjyS_1645000244) by smtp.aliyun-inc.com(127.0.0.1); Wed, 16 Feb 2022 16:30:45 +0800 From: Xin Hao To: sj@kernel.org Cc: xhao@linux.alibaba.com, rongwei.wang@linux.alibaba.com, akpm@linux-foundation.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH V1 1/5] mm/damon: Add NUMA local and remote variables in 'damon_region' Date: Wed, 16 Feb 2022 16:30:37 +0800 Message-Id: <2fb03665b39d7e3b222955ff690d73fe8e201c24.1645024354.git.xhao@linux.alibaba.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The purpose of adding these two variables 'local' & 'remote' is to obtain the struct 'damon_region' numa access status. Signed-off-by: Xin Hao Signed-off-by: Rongwei Wang --- include/linux/damon.h | 4 ++++ mm/damon/core.c | 6 ++++++ 2 files changed, 10 insertions(+) diff --git a/include/linux/damon.h b/include/linux/damon.h index 5e1e3a128b77..77d0937dcab5 100644 --- a/include/linux/damon.h +++ b/include/linux/damon.h @@ -41,6 +41,8 @@ struct damon_addr_range { * @nr_accesses: Access frequency of this region. * @list: List head for siblings. * @age: Age of this region. + * @local: Local numa node accesses. + * @remote: Remote numa node accesses. * * @age is initially zero, increased for each aggregation interval, and re= set * to zero again if the access frequency is significantly changed. If two @@ -56,6 +58,8 @@ struct damon_region { unsigned int age; /* private: Internal value for age calculation. */ unsigned int last_nr_accesses; + unsigned long local; + unsigned long remote; }; =20 /** diff --git a/mm/damon/core.c b/mm/damon/core.c index 1dd153c31c9e..933ef51afa71 100644 --- a/mm/damon/core.c +++ b/mm/damon/core.c @@ -45,6 +45,8 @@ struct damon_region *damon_new_region(unsigned long start= , unsigned long end) =20 region->age =3D 0; region->last_nr_accesses =3D 0; + region->local =3D 0; + region->remote =3D 0; =20 return region; } @@ -740,6 +742,8 @@ static void damon_merge_two_regions(struct damon_target= *t, =20 l->nr_accesses =3D (l->nr_accesses * sz_l + r->nr_accesses * sz_r) / (sz_l + sz_r); + l->remote =3D (l->remote * sz_l + r->remote * sz_r) / (sz_l + sz_r); + l->local =3D (l->local * sz_l + r->local * sz_r) / (sz_l + sz_r); l->age =3D (l->age * sz_l + r->age * sz_r) / (sz_l + sz_r); l->ar.end =3D r->ar.end; damon_destroy_region(r, t); @@ -812,6 +816,8 @@ static void damon_split_region_at(struct damon_ctx *ctx, =20 new->age =3D r->age; new->last_nr_accesses =3D r->last_nr_accesses; + new->local =3D r->local; + new->remote =3D r->remote; =20 damon_insert_region(new, r, damon_next_region(r), t); } --=20 2.27.0 From nobody Sun Jun 28 01:48:40 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E6E7C433FE for ; Wed, 16 Feb 2022 08:30:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230339AbiBPIbJ (ORCPT ); Wed, 16 Feb 2022 03:31:09 -0500 Received: from gmail-smtp-in.l.google.com ([23.128.96.19]:34978 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229526AbiBPIbA (ORCPT ); Wed, 16 Feb 2022 03:31:00 -0500 Received: from out30-56.freemail.mail.aliyun.com (out30-56.freemail.mail.aliyun.com [115.124.30.56]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 97AB821E31 for ; Wed, 16 Feb 2022 00:30:48 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R191e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04395;MF=xhao@linux.alibaba.com;NM=1;PH=DS;RN=6;SR=0;TI=SMTPD_---0V4cXjyl_1645000245; Received: from localhost.localdomain(mailfrom:xhao@linux.alibaba.com fp:SMTPD_---0V4cXjyl_1645000245) by smtp.aliyun-inc.com(127.0.0.1); Wed, 16 Feb 2022 16:30:46 +0800 From: Xin Hao To: sj@kernel.org Cc: xhao@linux.alibaba.com, rongwei.wang@linux.alibaba.com, akpm@linux-foundation.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH V1 2/5] mm/damon: Add 'damon_region' NUMA fault simulation support Date: Wed, 16 Feb 2022 16:30:38 +0800 Message-Id: <35c8c45267c6f2f5b6ec3559592342685106d39e.1645024354.git.xhao@linux.alibaba.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" These codes development here refers to NUMA balance code, it will cause a page_fault, in do_numa_page(), we will count 'damon_region' NUMA local and remote values. Signed-off-by: Xin Hao Signed-off-by: Rongwei Wang --- mm/damon/paddr.c | 23 +++++++++++++++++---- mm/damon/prmtv-common.c | 44 +++++++++++++++++++++++++++++++++++++++++ mm/damon/prmtv-common.h | 3 +++ mm/damon/vaddr.c | 32 +++++++++++++++++++++--------- 4 files changed, 89 insertions(+), 13 deletions(-) diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c index 5e8244f65a1a..b8feacf15592 100644 --- a/mm/damon/paddr.c +++ b/mm/damon/paddr.c @@ -16,9 +16,10 @@ #include "../internal.h" #include "prmtv-common.h" =20 -static bool __damon_pa_mkold(struct page *page, struct vm_area_struct *vma, +static bool __damon_pa_mk_set(struct page *page, struct vm_area_struct *vm= a, unsigned long addr, void *arg) { + bool result =3D false; struct page_vma_mapped_walk pvmw =3D { .page =3D page, .vma =3D vma, @@ -27,10 +28,24 @@ static bool __damon_pa_mkold(struct page *page, struct = vm_area_struct *vma, =20 while (page_vma_mapped_walk(&pvmw)) { addr =3D pvmw.address; - if (pvmw.pte) + if (pvmw.pte) { damon_ptep_mkold(pvmw.pte, vma->vm_mm, addr); - else + if (nr_online_nodes > 1) { + result =3D damon_ptep_mknone(pvmw.pte, vma, addr); + if (result) + flush_tlb_page(vma, addr); + } + } else { damon_pmdp_mkold(pvmw.pmd, vma->vm_mm, addr); + if (nr_online_nodes > 1) { + result =3D damon_pmdp_mknone(pvmw.pmd, vma, addr); + if (result) { + unsigned long haddr =3D addr & HPAGE_PMD_MASK; + + flush_tlb_range(vma, haddr, haddr + HPAGE_PMD_SIZE); + } + } + } } return true; } @@ -39,7 +54,7 @@ static void damon_pa_mkold(unsigned long paddr) { struct page *page =3D damon_get_page(PHYS_PFN(paddr)); struct rmap_walk_control rwc =3D { - .rmap_one =3D __damon_pa_mkold, + .rmap_one =3D __damon_pa_mk_set, .anon_lock =3D page_lock_anon_vma_read, }; bool need_lock; diff --git a/mm/damon/prmtv-common.c b/mm/damon/prmtv-common.c index 92a04f5831d6..35ac50fdf7b6 100644 --- a/mm/damon/prmtv-common.c +++ b/mm/damon/prmtv-common.c @@ -12,6 +12,50 @@ =20 #include "prmtv-common.h" =20 +bool damon_ptep_mknone(pte_t *pte, struct vm_area_struct *vma, unsigned lo= ng addr) +{ + pte_t oldpte, ptent; + bool preserve_write; + + oldpte =3D *pte; + if (pte_protnone(oldpte)) + return false; + + if (pte_present(oldpte)) { + preserve_write =3D pte_write(oldpte); + oldpte =3D ptep_modify_prot_start(vma, addr, pte); + ptent =3D pte_modify(oldpte, PAGE_NONE); + + if (preserve_write) + ptent =3D pte_mk_savedwrite(ptent); + + ptep_modify_prot_commit(vma, addr, pte, oldpte, ptent); + return true; + } + return false; +} + +bool damon_pmdp_mknone(pmd_t *pmd, struct vm_area_struct *vma, unsigned lo= ng addr) +{ + bool preserve_write; + pmd_t entry =3D *pmd; + + if (is_huge_zero_pmd(entry) || pmd_protnone(entry)) + return false; + + if (pmd_present(entry)) { + preserve_write =3D pmd_write(entry); + entry =3D pmdp_invalidate(vma, addr, pmd); + entry =3D pmd_modify(entry, PAGE_NONE); + if (preserve_write) + entry =3D pmd_mk_savedwrite(entry); + + set_pmd_at(vma->vm_mm, addr, pmd, entry); + return true; + } + return false; +} + /* * Get an online page for a pfn if it's in the LRU list. Otherwise, retur= ns * NULL. diff --git a/mm/damon/prmtv-common.h b/mm/damon/prmtv-common.h index e790cb5f8fe0..002a308facd0 100644 --- a/mm/damon/prmtv-common.h +++ b/mm/damon/prmtv-common.h @@ -7,6 +7,9 @@ =20 #include =20 +bool damon_ptep_mknone(pte_t *pte, struct vm_area_struct *vma, unsigned lo= ng addr); +bool damon_pmdp_mknone(pmd_t *pmd, struct vm_area_struct *vma, unsigned lo= ng addr); + struct page *damon_get_page(unsigned long pfn); =20 void damon_ptep_mkold(pte_t *pte, struct mm_struct *mm, unsigned long addr= ); diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index 89b6468da2b9..732b41ed134c 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -367,9 +367,10 @@ static void damon_va_update(struct damon_ctx *ctx) } } =20 -static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned long addr, +static int damon_va_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long next, struct mm_walk *walk) { + bool result =3D false; pte_t *pte; spinlock_t *ptl; =20 @@ -377,7 +378,14 @@ static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned = long addr, ptl =3D pmd_lock(walk->mm, pmd); if (pmd_huge(*pmd)) { damon_pmdp_mkold(pmd, walk->mm, addr); + if (nr_online_nodes > 1) + result =3D damon_pmdp_mknone(pmd, walk->vma, addr); spin_unlock(ptl); + if (result) { + unsigned long haddr =3D addr & HPAGE_PMD_MASK; + + flush_tlb_range(walk->vma, haddr, haddr + HPAGE_PMD_SIZE); + } return 0; } spin_unlock(ptl); @@ -386,11 +394,17 @@ static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned= long addr, if (pmd_none(*pmd) || unlikely(pmd_bad(*pmd))) return 0; pte =3D pte_offset_map_lock(walk->mm, pmd, addr, &ptl); - if (!pte_present(*pte)) - goto out; + if (!pte_present(*pte)) { + pte_unmap_unlock(pte, ptl); + return 0; + } damon_ptep_mkold(pte, walk->mm, addr); -out: + if (nr_online_nodes > 1) + result =3D damon_ptep_mknone(pte, walk->vma, addr); pte_unmap_unlock(pte, ptl); + if (result) + flush_tlb_page(walk->vma, addr); + return 0; } =20 @@ -450,15 +464,15 @@ static int damon_mkold_hugetlb_entry(pte_t *pte, unsi= gned long hmask, #define damon_mkold_hugetlb_entry NULL #endif /* CONFIG_HUGETLB_PAGE */ =20 -static const struct mm_walk_ops damon_mkold_ops =3D { - .pmd_entry =3D damon_mkold_pmd_entry, +static const struct mm_walk_ops damon_va_ops =3D { + .pmd_entry =3D damon_va_pmd_entry, .hugetlb_entry =3D damon_mkold_hugetlb_entry, }; =20 -static void damon_va_mkold(struct mm_struct *mm, unsigned long addr) +static void damon_va_check(struct mm_struct *mm, unsigned long addr) { mmap_read_lock(mm); - walk_page_range(mm, addr, addr + 1, &damon_mkold_ops, NULL); + walk_page_range(mm, addr, addr + 1, &damon_va_ops, NULL); mmap_read_unlock(mm); } =20 @@ -471,7 +485,7 @@ static void __damon_va_prepare_access_check(struct damo= n_ctx *ctx, { r->sampling_addr =3D damon_rand(r->ar.start, r->ar.end); =20 - damon_va_mkold(mm, r->sampling_addr); + damon_va_check(mm, r->sampling_addr); } =20 static void damon_va_prepare_access_checks(struct damon_ctx *ctx) --=20 2.27.0 From nobody Sun Jun 28 01:48:40 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C636C433F5 for ; Wed, 16 Feb 2022 08:31:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231208AbiBPIbR (ORCPT ); Wed, 16 Feb 2022 03:31:17 -0500 Received: from gmail-smtp-in.l.google.com ([23.128.96.19]:35448 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229908AbiBPIbD (ORCPT ); Wed, 16 Feb 2022 03:31:03 -0500 Received: from out30-131.freemail.mail.aliyun.com (out30-131.freemail.mail.aliyun.com [115.124.30.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ADC822A4A02 for ; Wed, 16 Feb 2022 00:30:49 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R211e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04423;MF=xhao@linux.alibaba.com;NM=1;PH=DS;RN=6;SR=0;TI=SMTPD_---0V4cXjyv_1645000246; Received: from localhost.localdomain(mailfrom:xhao@linux.alibaba.com fp:SMTPD_---0V4cXjyv_1645000246) by smtp.aliyun-inc.com(127.0.0.1); Wed, 16 Feb 2022 16:30:47 +0800 From: Xin Hao To: sj@kernel.org Cc: xhao@linux.alibaba.com, rongwei.wang@linux.alibaba.com, akpm@linux-foundation.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH V1 3/5] mm/damon: Add 'damon_region' NUMA access statistics core implementation Date: Wed, 16 Feb 2022 16:30:39 +0800 Message-Id: X-Mailer: git-send-email 2.31.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" After setting PTE none or PMD none in DAMON, NUMA access of "damon_region" = will be counted in page fault if current pid matches the pid that DAMON is monitori= ng. Signed-off-by: Xin Hao Signed-off-by: Rongwei Wang --- include/linux/damon.h | 18 ++++++++++ mm/damon/core.c | 80 +++++++++++++++++++++++++++++++++++++++++-- mm/damon/dbgfs.c | 18 +++++++--- mm/damon/vaddr.c | 11 ++---- mm/huge_memory.c | 5 +++ mm/memory.c | 5 +++ 6 files changed, 121 insertions(+), 16 deletions(-) diff --git a/include/linux/damon.h b/include/linux/damon.h index 77d0937dcab5..5bf1eb92584b 100644 --- a/include/linux/damon.h +++ b/include/linux/damon.h @@ -12,12 +12,16 @@ #include #include #include +#include =20 /* Minimal region size. Every damon_region is aligned by this. */ #define DAMON_MIN_REGION PAGE_SIZE /* Max priority score for DAMON-based operation schemes */ #define DAMOS_MAX_SCORE (99) =20 +extern struct damon_ctx **dbgfs_ctxs; +extern int dbgfs_nr_ctxs; + /* Get a random number in [l, r) */ static inline unsigned long damon_rand(unsigned long l, unsigned long r) { @@ -68,6 +72,7 @@ struct damon_region { * @nr_regions: Number of monitoring target regions of this target. * @regions_list: Head of the monitoring target regions of this target. * @list: List head for siblings. + * @target_lock: Use damon_region lock to avoid race. * * Each monitoring context could have multiple targets. For example, a co= ntext * for virtual memory address spaces could have multiple target processes.= The @@ -80,6 +85,7 @@ struct damon_target { unsigned int nr_regions; struct list_head regions_list; struct list_head list; + spinlock_t target_lock; }; =20 /** @@ -503,8 +509,20 @@ int damon_stop(struct damon_ctx **ctxs, int nr_ctxs); #endif /* CONFIG_DAMON */ =20 #ifdef CONFIG_DAMON_VADDR + +/* + * 't->id' should be the pointer to the relevant 'struct pid' having refer= ence + * count. Caller must put the returned task, unless it is NULL. + */ +static inline struct task_struct *damon_get_task_struct(struct damon_targe= t *t) +{ + return get_pid_task((struct pid *)t->id, PIDTYPE_PID); +} bool damon_va_target_valid(void *t); void damon_va_set_primitives(struct damon_ctx *ctx); +void damon_numa_fault(int page_nid, int node_id, struct vm_fault *vmf); +#else +static inline void damon_numa_fault(int page_nid, int node_id, struct vm_f= ault *vmf) { } #endif /* CONFIG_DAMON_VADDR */ =20 #ifdef CONFIG_DAMON_PADDR diff --git a/mm/damon/core.c b/mm/damon/core.c index 933ef51afa71..970fc02abeba 100644 --- a/mm/damon/core.c +++ b/mm/damon/core.c @@ -157,6 +157,7 @@ struct damon_target *damon_new_target(unsigned long id) t->id =3D id; t->nr_regions =3D 0; INIT_LIST_HEAD(&t->regions_list); + spin_lock_init(&t->target_lock); =20 return t; } @@ -792,8 +793,11 @@ static void kdamond_merge_regions(struct damon_ctx *c,= unsigned int threshold, { struct damon_target *t; =20 - damon_for_each_target(t, c) + damon_for_each_target(t, c) { + spin_lock(&t->target_lock); damon_merge_regions_of(t, threshold, sz_limit); + spin_unlock(&t->target_lock); + } } =20 /* @@ -879,8 +883,11 @@ static void kdamond_split_regions(struct damon_ctx *ct= x) nr_regions < ctx->max_nr_regions / 3) nr_subregions =3D 3; =20 - damon_for_each_target(t, ctx) + damon_for_each_target(t, ctx) { + spin_lock(&t->target_lock); damon_split_regions_of(ctx, t, nr_subregions); + spin_unlock(&t->target_lock); + } =20 last_nr_regions =3D nr_regions; } @@ -1000,6 +1007,73 @@ static int kdamond_wait_activation(struct damon_ctx = *ctx) return -EBUSY; } =20 +static struct damon_target *get_damon_target(struct task_struct *task) +{ + int i; + unsigned long id1, id2; + struct damon_target *t; + + rcu_read_lock(); + for (i =3D 0; i < READ_ONCE(dbgfs_nr_ctxs); i++) { + struct damon_ctx *ctx =3D rcu_dereference(dbgfs_ctxs[i]); + + if (!ctx || !ctx->kdamond) + continue; + damon_for_each_target(t, dbgfs_ctxs[i]) { + struct task_struct *ts =3D damon_get_task_struct(t); + + if (ts) { + id1 =3D (unsigned long)pid_vnr((struct pid *)t->id); + id2 =3D (unsigned long)pid_vnr(get_task_pid(task, PIDTYPE_PID)); + put_task_struct(ts); + if (id1 =3D=3D id2) + return t; + } + } + } + rcu_read_unlock(); + + return NULL; +} + +static struct damon_region *get_damon_region(struct damon_target *t, unsig= ned long addr) +{ + struct damon_region *r, *next; + + if (!t || !addr) + return NULL; + + spin_lock(&t->target_lock); + damon_for_each_region_safe(r, next, t) { + if (r->ar.start <=3D addr && r->ar.end >=3D addr) { + spin_unlock(&t->target_lock); + return r; + } + } + spin_unlock(&t->target_lock); + + return NULL; +} + +void damon_numa_fault(int page_nid, int node_id, struct vm_fault *vmf) +{ + struct damon_target *t; + struct damon_region *r; + + if (nr_online_nodes > 1) { + t =3D get_damon_target(current); + if (!t) + return; + r =3D get_damon_region(t, vmf->address); + if (r) { + if (page_nid =3D=3D node_id) + r->local++; + else + r->remote++; + } + } +} + /* * The monitoring daemon that runs as a kernel thread */ @@ -1057,8 +1131,10 @@ static int kdamond_fn(void *data) } } damon_for_each_target(t, ctx) { + spin_lock(&t->target_lock); damon_for_each_region_safe(r, next, t) damon_destroy_region(r, t); + spin_unlock(&t->target_lock); } =20 if (ctx->callback.before_terminate) diff --git a/mm/damon/dbgfs.c b/mm/damon/dbgfs.c index 5b899601e56c..c7f4e95abc14 100644 --- a/mm/damon/dbgfs.c +++ b/mm/damon/dbgfs.c @@ -15,11 +15,12 @@ #include #include =20 -static struct damon_ctx **dbgfs_ctxs; -static int dbgfs_nr_ctxs; +struct damon_ctx **dbgfs_ctxs; +int dbgfs_nr_ctxs; static struct dentry **dbgfs_dirs; static DEFINE_MUTEX(damon_dbgfs_lock); =20 + /* * Returns non-empty string on success, negative error code otherwise. */ @@ -808,10 +809,18 @@ static int dbgfs_rm_context(char *name) return -ENOMEM; } =20 - for (i =3D 0, j =3D 0; i < dbgfs_nr_ctxs; i++) { + dbgfs_nr_ctxs--; + /* Prevent NUMA fault get the wrong value */ + smp_mb(); + + for (i =3D 0, j =3D 0; i < dbgfs_nr_ctxs + 1; i++) { if (dbgfs_dirs[i] =3D=3D dir) { + struct damon_ctx *tmp_ctx =3D dbgfs_ctxs[i]; + + rcu_assign_pointer(dbgfs_ctxs[i], NULL); + synchronize_rcu(); debugfs_remove(dbgfs_dirs[i]); - dbgfs_destroy_ctx(dbgfs_ctxs[i]); + dbgfs_destroy_ctx(tmp_ctx); continue; } new_dirs[j] =3D dbgfs_dirs[i]; @@ -823,7 +832,6 @@ static int dbgfs_rm_context(char *name) =20 dbgfs_dirs =3D new_dirs; dbgfs_ctxs =3D new_ctxs; - dbgfs_nr_ctxs--; =20 return 0; } diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index 732b41ed134c..78b90972d171 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -22,15 +22,6 @@ #define DAMON_MIN_REGION 1 #endif =20 -/* - * 't->id' should be the pointer to the relevant 'struct pid' having refer= ence - * count. Caller must put the returned task, unless it is NULL. - */ -static inline struct task_struct *damon_get_task_struct(struct damon_targe= t *t) -{ - return get_pid_task((struct pid *)t->id, PIDTYPE_PID); -} - /* * Get the mm_struct of the given target * @@ -363,7 +354,9 @@ static void damon_va_update(struct damon_ctx *ctx) damon_for_each_target(t, ctx) { if (damon_va_three_regions(t, three_regions)) continue; + spin_lock(&t->target_lock); damon_va_apply_three_regions(t, three_regions); + spin_unlock(&t->target_lock); } } =20 diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 406a3c28c026..9cb413a8cd4a 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -34,6 +34,7 @@ #include #include #include +#include =20 #include #include @@ -1450,6 +1451,10 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vm= f) flags |=3D TNF_NO_GROUP; =20 page_nid =3D page_to_nid(page); + + /* Get the NUMA accesses of monitored processes by DAMON */ + damon_numa_fault(page_nid, numa_node_id(), vmf); + last_cpupid =3D page_cpupid_last(page); target_nid =3D numa_migrate_prep(page, vma, haddr, page_nid, &flags); diff --git a/mm/memory.c b/mm/memory.c index c125c4969913..fb55264f36af 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -74,6 +74,7 @@ #include #include #include +#include =20 #include =20 @@ -4392,6 +4393,10 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) =20 last_cpupid =3D page_cpupid_last(page); page_nid =3D page_to_nid(page); + + /* Get the NUMA accesses of monitored processes by DAMON */ + damon_numa_fault(page_nid, numa_node_id(), vmf); + target_nid =3D numa_migrate_prep(page, vma, vmf->address, page_nid, &flags); if (target_nid =3D=3D NUMA_NO_NODE) { --=20 2.27.0 From nobody Sun Jun 28 01:48:40 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6486BC433F5 for ; Wed, 16 Feb 2022 08:31:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230254AbiBPIbO (ORCPT ); Wed, 16 Feb 2022 03:31:14 -0500 Received: from gmail-smtp-in.l.google.com ([23.128.96.19]:35450 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229958AbiBPIbD (ORCPT ); Wed, 16 Feb 2022 03:31:03 -0500 Received: from out30-54.freemail.mail.aliyun.com (out30-54.freemail.mail.aliyun.com [115.124.30.54]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 83F8222BF7 for ; Wed, 16 Feb 2022 00:30:50 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R241e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04400;MF=xhao@linux.alibaba.com;NM=1;PH=DS;RN=6;SR=0;TI=SMTPD_---0V4cXjz7_1645000247; Received: from localhost.localdomain(mailfrom:xhao@linux.alibaba.com fp:SMTPD_---0V4cXjz7_1645000247) by smtp.aliyun-inc.com(127.0.0.1); Wed, 16 Feb 2022 16:30:48 +0800 From: Xin Hao To: sj@kernel.org Cc: xhao@linux.alibaba.com, rongwei.wang@linux.alibaba.com, akpm@linux-foundation.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH V1 4/5] mm/damon/dbgfs: Add numa simulate switch Date: Wed, 16 Feb 2022 16:30:40 +0800 Message-Id: <20b2e1e1a60431e7d0f47df3ef9619db3bda2946.1645024354.git.xhao@linux.alibaba.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" For applications that frequently access the memory area, Doing numa simulation will cause a lot of pagefault and tlb misses which will cause the applications performance regression. So there adds a switch to turn off the numa simulation function by default, if you want to turn on this function just do like below. # cd /sys/kernel/debug/damon/ # echo on > numa_stat # cat numa_stat # on Signed-off-by: Xin Hao Signed-off-by: Rongwei Wang --- include/linux/damon.h | 3 +++ mm/damon/core.c | 10 ++++++++- mm/damon/dbgfs.c | 52 +++++++++++++++++++++++++++++++++++++++++-- mm/damon/paddr.c | 6 +++-- mm/damon/vaddr.c | 6 +++-- 5 files changed, 70 insertions(+), 7 deletions(-) diff --git a/include/linux/damon.h b/include/linux/damon.h index 5bf1eb92584b..c7d7613e1a17 100644 --- a/include/linux/damon.h +++ b/include/linux/damon.h @@ -19,6 +19,9 @@ /* Max priority score for DAMON-based operation schemes */ #define DAMOS_MAX_SCORE (99) =20 +/* Switch for NUMA fault */ +DECLARE_STATIC_KEY_FALSE(numa_stat_enabled_key); + extern struct damon_ctx **dbgfs_ctxs; extern int dbgfs_nr_ctxs; =20 diff --git a/mm/damon/core.c b/mm/damon/core.c index 970fc02abeba..4aa3c2d3895c 100644 --- a/mm/damon/core.c +++ b/mm/damon/core.c @@ -1060,7 +1060,8 @@ void damon_numa_fault(int page_nid, int node_id, stru= ct vm_fault *vmf) struct damon_target *t; struct damon_region *r; =20 - if (nr_online_nodes > 1) { + if (static_branch_unlikely(&numa_stat_enabled_key) + && nr_online_nodes > 1) { t =3D get_damon_target(current); if (!t) return; @@ -1151,6 +1152,13 @@ static int kdamond_fn(void *data) nr_running_ctxs--; mutex_unlock(&damon_lock); =20 + /* + * when no kdamond threads are running, the + * 'numa_stat_enabled_key' keeps default value. + */ + if (!nr_running_ctxs) + static_branch_disable(&numa_stat_enabled_key); + return 0; } =20 diff --git a/mm/damon/dbgfs.c b/mm/damon/dbgfs.c index c7f4e95abc14..0ef35dbfda39 100644 --- a/mm/damon/dbgfs.c +++ b/mm/damon/dbgfs.c @@ -609,6 +609,49 @@ static ssize_t dbgfs_kdamond_pid_read(struct file *fil= e, return len; } =20 +DEFINE_STATIC_KEY_FALSE(numa_stat_enabled_key); + +static ssize_t dbgfs_numa_stat_read(struct file *file, + char __user *buf, size_t count, loff_t *ppos) +{ + char numa_on_buf[5]; + bool enable =3D static_branch_unlikely(&numa_stat_enabled_key); + int len; + + len =3D scnprintf(numa_on_buf, 5, enable ? "on\n" : "off\n"); + + return simple_read_from_buffer(buf, count, ppos, numa_on_buf, len); +} + +static ssize_t dbgfs_numa_stat_write(struct file *file, + const char __user *buf, size_t count, loff_t *ppos) +{ + ssize_t ret =3D 0; + char *kbuf; + + kbuf =3D user_input_str(buf, count, ppos); + if (IS_ERR(kbuf)) + return PTR_ERR(kbuf); + + /* Remove white space */ + if (sscanf(kbuf, "%s", kbuf) !=3D 1) { + kfree(kbuf); + return -EINVAL; + } + + if (!strncmp(kbuf, "on", count)) + static_branch_enable(&numa_stat_enabled_key); + else if (!strncmp(kbuf, "off", count)) + static_branch_disable(&numa_stat_enabled_key); + else + ret =3D -EINVAL; + + if (!ret) + ret =3D count; + kfree(kbuf); + return ret; +} + static int damon_dbgfs_open(struct inode *inode, struct file *file) { file->private_data =3D inode->i_private; @@ -645,12 +688,17 @@ static const struct file_operations kdamond_pid_fops = =3D { .read =3D dbgfs_kdamond_pid_read, }; =20 +static const struct file_operations numa_stat_ops =3D { + .write =3D dbgfs_numa_stat_write, + .read =3D dbgfs_numa_stat_read, +}; + static void dbgfs_fill_ctx_dir(struct dentry *dir, struct damon_ctx *ctx) { const char * const file_names[] =3D {"attrs", "schemes", "target_ids", - "init_regions", "kdamond_pid"}; + "init_regions", "kdamond_pid", "numa_stat"}; const struct file_operations *fops[] =3D {&attrs_fops, &schemes_fops, - &target_ids_fops, &init_regions_fops, &kdamond_pid_fops}; + &target_ids_fops, &init_regions_fops, &kdamond_pid_fops, &numa_stat_ops}; int i; =20 for (i =3D 0; i < ARRAY_SIZE(file_names); i++) diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c index b8feacf15592..9b9920784f22 100644 --- a/mm/damon/paddr.c +++ b/mm/damon/paddr.c @@ -30,14 +30,16 @@ static bool __damon_pa_mk_set(struct page *page, struct= vm_area_struct *vma, addr =3D pvmw.address; if (pvmw.pte) { damon_ptep_mkold(pvmw.pte, vma->vm_mm, addr); - if (nr_online_nodes > 1) { + if (static_branch_unlikely(&numa_stat_enabled_key) && + nr_online_nodes > 1) { result =3D damon_ptep_mknone(pvmw.pte, vma, addr); if (result) flush_tlb_page(vma, addr); } } else { damon_pmdp_mkold(pvmw.pmd, vma->vm_mm, addr); - if (nr_online_nodes > 1) { + if (static_branch_unlikely(&numa_stat_enabled_key) && + nr_online_nodes > 1) { result =3D damon_pmdp_mknone(pvmw.pmd, vma, addr); if (result) { unsigned long haddr =3D addr & HPAGE_PMD_MASK; diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index 78b90972d171..5c2e2c2e29bb 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -371,7 +371,8 @@ static int damon_va_pmd_entry(pmd_t *pmd, unsigned long= addr, ptl =3D pmd_lock(walk->mm, pmd); if (pmd_huge(*pmd)) { damon_pmdp_mkold(pmd, walk->mm, addr); - if (nr_online_nodes > 1) + if (static_branch_unlikely(&numa_stat_enabled_key) && + nr_online_nodes > 1) result =3D damon_pmdp_mknone(pmd, walk->vma, addr); spin_unlock(ptl); if (result) { @@ -392,7 +393,8 @@ static int damon_va_pmd_entry(pmd_t *pmd, unsigned long= addr, return 0; } damon_ptep_mkold(pte, walk->mm, addr); - if (nr_online_nodes > 1) + if (static_branch_unlikely(&numa_stat_enabled_key) && + nr_online_nodes > 1) result =3D damon_ptep_mknone(pte, walk->vma, addr); pte_unmap_unlock(pte, ptl); if (result) --=20 2.27.0 From nobody Sun Jun 28 01:48:40 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 187F6C433EF for ; Wed, 16 Feb 2022 08:31:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231298AbiBPIbT (ORCPT ); Wed, 16 Feb 2022 03:31:19 -0500 Received: from gmail-smtp-in.l.google.com ([23.128.96.19]:35782 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229996AbiBPIbF (ORCPT ); Wed, 16 Feb 2022 03:31:05 -0500 Received: from out30-131.freemail.mail.aliyun.com (out30-131.freemail.mail.aliyun.com [115.124.30.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E76782A417B for ; Wed, 16 Feb 2022 00:30:51 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R121e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04400;MF=xhao@linux.alibaba.com;NM=1;PH=DS;RN=6;SR=0;TI=SMTPD_---0V4cXjzP_1645000248; Received: from localhost.localdomain(mailfrom:xhao@linux.alibaba.com fp:SMTPD_---0V4cXjzP_1645000248) by smtp.aliyun-inc.com(127.0.0.1); Wed, 16 Feb 2022 16:30:49 +0800 From: Xin Hao To: sj@kernel.org Cc: xhao@linux.alibaba.com, rongwei.wang@linux.alibaba.com, akpm@linux-foundation.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH V1 5/5] mm/damon/tracepoint: Add 'damon_region' NUMA access statistics support Date: Wed, 16 Feb 2022 16:30:41 +0800 Message-Id: X-Mailer: git-send-email 2.31.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This patch is used to support 'damon_region' NUMA access for tracepoint, The purpose of this is to facilitate users to obtain the numa access status of 'damon_region' through perf or damo tools. Signed-off-by: Xin Hao Signed-off-by: Rongwei Wang --- include/trace/events/damon.h | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/include/trace/events/damon.h b/include/trace/events/damon.h index c79f1d4c39af..687b7aba751e 100644 --- a/include/trace/events/damon.h +++ b/include/trace/events/damon.h @@ -23,6 +23,8 @@ TRACE_EVENT(damon_aggregated, __field(unsigned long, end) __field(unsigned int, nr_accesses) __field(unsigned int, age) + __field(unsigned long, local) + __field(unsigned long, remote) ), =20 TP_fast_assign( @@ -32,12 +34,15 @@ TRACE_EVENT(damon_aggregated, __entry->end =3D r->ar.end; __entry->nr_accesses =3D r->nr_accesses; __entry->age =3D r->age; + __entry->local =3D r->local; + __entry->remote =3D r->remote; ), =20 - TP_printk("target_id=3D%lu nr_regions=3D%u %lu-%lu: %u %u", + TP_printk("target_id=3D%lu nr_regions=3D%u %lu-%lu: %u %u %lu %lu", __entry->target_id, __entry->nr_regions, __entry->start, __entry->end, - __entry->nr_accesses, __entry->age) + __entry->nr_accesses, __entry->age, + __entry->local, __entry->remote) ); =20 #endif /* _TRACE_DAMON_H */ --=20 2.27.0