From nobody Tue Nov 26 08:47:30 2024 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8A6381BDA9C for ; Sat, 19 Oct 2024 01:29:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729301388; cv=none; b=Yj5xTDxS8SiHzFWPF6llOrZ3ujwAQs4mVUDN4kSTLh9u4nSF2CoCMvoBdXYFvwIsowmEkNeNJwAxuSP5b+BNv3OS5m29Vvjz+F7SEN7zfH1QF9yNhejUi/fdNb8cvape9OnPOnyvn2HKlt83ndf24V60WhE6vVVJ0Dk2+KSZcAQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729301388; c=relaxed/simple; bh=ADHj64695AnmWs6scDXXAuyNjPNjewRXryZ8+dKDUIo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=YegSYNwEfjCAQ+KqRGmuUUt1HCMu3kFWvzTOmp5Q1Sa1TwKl0GjhZn6F0s1ENDg9AFQufMqVyoHZ4kYAMuZTkGW7/Wg4hEEWD1qGOfEKfjjE0j2ssAgRKA+SFjaq9XS7cRhcr26hwOLlgt7sdvxAvcAuPpvAKOuwWGvOFth8cxQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jthoughton.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=viXTgd/+; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jthoughton.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="viXTgd/+" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6e1fbe2a6b1so48671107b3.2 for ; Fri, 18 Oct 2024 18:29:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729301385; x=1729906185; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=JWmoU09y+6wa+3R3ifZVkoogK5Ofgf3reUuHP39SGJ0=; b=viXTgd/+78WBIA+cBXcXXn9PPKzGgZPL3VXpi9qdNKMmmXitaVomB0T4MCr9+IEzGC vP4IDaPubm2fW4tWyAQyuPctcYAbdWz0aCMaOXBPR5vGob8/keahmg7F1sbDrBYUBfu8 5xMqa29W5f8VjBAdTdvh4vGMisq63G45GMCPeOn5x2qxrA6W/Q+3RJXCHHS/CE7r+oBu Co4AgCDiZy8tPkSukpb3LxbZKBdB/v0EIxEhQNNklVjrLdzF/v3SE1FnJtp0mqjNxmCo lf/q/TuZ65fHNjXFUDo3HdaAzq3l1kfB1lNXmA6Zco7jMOMKvCAdi0cgYI08ET5ZTKx5 WMdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729301385; x=1729906185; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=JWmoU09y+6wa+3R3ifZVkoogK5Ofgf3reUuHP39SGJ0=; b=ZvvZkmukycUCNwCF84C0IMTlpf9zjUjxibhqIqXBDwm98QYkyXhby1k/jYwO/vl9PQ BUFoxltWG1bp8HwOTWtFK/ESySejL11obNbya3kxHIifxsYrVXZhxiDbDiDjZfPrjk2J 12zPqI82uxV+YO2o9gCcGAz6aFCOdp3llRuQ8dz7e/9B0JYcWmBcoL/CeUPwmrj6GPY8 0uiuFPn9WbwcQLj45uO0WSFAUejwi9HJrpTrn4X2rRrieSYGHhndJ2KFJ96RpPbbgVcR UNj3pv25+vLUS12zDwR6+GIR3nuUnYIktum4Te0iGfGwMM/nxCv84hs1fFRNzTucwLNH vN2w== X-Forwarded-Encrypted: i=1; AJvYcCW27JLYO2SkhWaEZaUtHmGVrNsHXoSAc0yCIi/M8KGnbw9Dl+zMxiHgiPCFE76PF8dh5j8qoZygd3mWD9s=@vger.kernel.org X-Gm-Message-State: AOJu0YzUXbohH7lEIBGpi960+XUGKVgdr6e1gELks7CpTxQFZYcx9cD/ hn885+TSBtwaNZ7X+bsQibPxVJM7BqTyWZDqYqBTDnhSvRLAvjU2sWi33P/1OBoqskSUeOJSazL SBTMSqvabO7NMr5VD2A== X-Google-Smtp-Source: AGHT+IE7mxydzjHIcaCtSw9GRjaWSidW3zTmTU0yiTMTLG+FkvhjpHmaauVlHv0Tv7yfhEo/zPT4VQCUKP70YtII X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:a05:690c:2e90:b0:6e3:1627:e866 with SMTP id 00721157ae682-6e5bfbd836bmr424387b3.3.1729301385447; Fri, 18 Oct 2024 18:29:45 -0700 (PDT) Date: Sat, 19 Oct 2024 01:29:38 +0000 In-Reply-To: <20241019012940.3656292-1-jthoughton@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241019012940.3656292-1-jthoughton@google.com> X-Mailer: git-send-email 2.47.0.105.g07ac214952-goog Message-ID: <20241019012940.3656292-2-jthoughton@google.com> Subject: [PATCH 1/2] mm: multi-gen LRU: remove MM_LEAF_OLD and MM_NONLEAF_TOTAL stats From: James Houghton To: Andrew Morton Cc: Sean Christopherson , Paolo Bonzini , David Matlack , David Rientjes , James Houghton , Oliver Upton , David Stevens , Yu Zhao , Wei Xu , Axel Rasmussen , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Yu Zhao The removed stats, MM_LEAF_OLD and MM_NONLEAF_TOTAL, are not very helpful and become more complicated to properly compute when adding test/clear_young() notifiers in MGLRU's mm walk. Signed-off-by: Yu Zhao Signed-off-by: James Houghton --- include/linux/mmzone.h | 2 -- mm/vmscan.c | 14 +++++--------- 2 files changed, 5 insertions(+), 11 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 96dea31fb211..691c635d8d1f 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -460,9 +460,7 @@ struct lru_gen_folio { =20 enum { MM_LEAF_TOTAL, /* total leaf entries */ - MM_LEAF_OLD, /* old leaf entries */ MM_LEAF_YOUNG, /* young leaf entries */ - MM_NONLEAF_TOTAL, /* total non-leaf entries */ MM_NONLEAF_FOUND, /* non-leaf entries found in Bloom filters */ MM_NONLEAF_ADDED, /* non-leaf entries added to Bloom filters */ NR_MM_STATS diff --git a/mm/vmscan.c b/mm/vmscan.c index 2d0486189804..60669f8bba46 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3405,7 +3405,6 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long = start, unsigned long end, continue; =20 if (!pte_young(ptent)) { - walk->mm_stats[MM_LEAF_OLD]++; continue; } =20 @@ -3558,7 +3557,6 @@ static void walk_pmd_range(pud_t *pud, unsigned long = start, unsigned long end, walk->mm_stats[MM_LEAF_TOTAL]++; =20 if (!pmd_young(val)) { - walk->mm_stats[MM_LEAF_OLD]++; continue; } =20 @@ -3570,8 +3568,6 @@ static void walk_pmd_range(pud_t *pud, unsigned long = start, unsigned long end, continue; } =20 - walk->mm_stats[MM_NONLEAF_TOTAL]++; - if (!walk->force_scan && should_clear_pmd_young()) { if (!pmd_young(val)) continue; @@ -5262,11 +5258,11 @@ static void lru_gen_seq_show_full(struct seq_file *= m, struct lruvec *lruvec, for (tier =3D 0; tier < MAX_NR_TIERS; tier++) { seq_printf(m, " %10d", tier); for (type =3D 0; type < ANON_AND_FILE; type++) { - const char *s =3D " "; + const char *s =3D "xxx"; unsigned long n[3] =3D {}; =20 if (seq =3D=3D max_seq) { - s =3D "RT "; + s =3D "RTx"; n[0] =3D READ_ONCE(lrugen->avg_refaulted[type][tier]); n[1] =3D READ_ONCE(lrugen->avg_total[type][tier]); } else if (seq =3D=3D min_seq[type] || NR_HIST_GENS > 1) { @@ -5288,14 +5284,14 @@ static void lru_gen_seq_show_full(struct seq_file *= m, struct lruvec *lruvec, =20 seq_puts(m, " "); for (i =3D 0; i < NR_MM_STATS; i++) { - const char *s =3D " "; + const char *s =3D "xxxx"; unsigned long n =3D 0; =20 if (seq =3D=3D max_seq && NR_HIST_GENS =3D=3D 1) { - s =3D "LOYNFA"; + s =3D "TYFA"; n =3D READ_ONCE(mm_state->stats[hist][i]); } else if (seq !=3D max_seq && NR_HIST_GENS > 1) { - s =3D "loynfa"; + s =3D "tyfa"; n =3D READ_ONCE(mm_state->stats[hist][i]); } =20 --=20 2.47.0.105.g07ac214952-goog From nobody Tue Nov 26 08:47:30 2024 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5662C1BE226 for ; Sat, 19 Oct 2024 01:29:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729301390; cv=none; b=PoqZWw3D8zcbqJoKBpXR7nRBIaRH9zIXn0XjQ81FRGhz0QbfuhToKVQLWmLNUlIXPkZCg+g8iItiDB/R9bHwmqqkBHs0robcBNUrDt0SI2gtJb0UsMjFPRGNxm1oItCm3Jhm5zFWbTYxFtIjsN5Xfb2qI0tlVSyIcOGDoZnNIqI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729301390; c=relaxed/simple; bh=QIezdTSRPHRnsz35ebM++m8bsNTvgEcRASfkNHbnPuA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=OlC6nZ9DeVhdW3YQdiSc+lswYdLF3V0lav00lOIhuCgvSVd0ICV2D7Rzy9ioISjNNmXe1nIT/mcs/XRjOdM4zoTzQ+TdCo9tLpsTy3bohpKug2Ig78FnIomLU/BLi1834b1LkFu7ZpPcCHWSFfp/gwPsWgQotn7roOSNDXzJ1Fk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jthoughton.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=wQv/w5u1; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jthoughton.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="wQv/w5u1" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6dbbeee08f0so51151677b3.0 for ; Fri, 18 Oct 2024 18:29:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729301387; x=1729906187; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=eKoy9FeSC1Po3W3xcBQFvk5zP5oePGJ83EILIsAFUYs=; b=wQv/w5u1Bys3IZKspZ1Hsg5160Cej5LQ+aDnmOjWIxMX9AAnIzd0PjM3ElfASiC5a1 qtXEf7VPCbexAI7BAhbJNsGD7b1UHEaMyWpTQZxqfC0pWhJeUW14jM0CafTunMJMjyyG Ekzo1X6Ny4pgoSckkHRxSfFcmMgvwE29yPyZrlemePF3LyAjvjHPSOjuwMPcxTboFtYJ tB/LKdB7IJaCaeBGC43CZ4IpC1HsBBY4QkDc1bYnl4Lw96KWfjaQLGhG6KhLcig3fogd hgfWIJBzsexCkqNnJfxgMVrAzozOKWnFUYrSzKWsj1rXsxYrQVwTteG6eU51Rgd6Lxxl foUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729301387; x=1729906187; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=eKoy9FeSC1Po3W3xcBQFvk5zP5oePGJ83EILIsAFUYs=; b=EZe2fCT3LFfQ74HQmeE4pjD/5TT99XFCcHQYdDFgT6CQlQ0TMb5OA2+T/oYy71O41U KRtka8KKjajFY/UKDOTcbKlWAC4YZ5d8ECWW63/LKvn0g/zPJUblAbIEWm7aee1YZTU1 1IS78dmon4okqeGlcGVpRCf00Cr7b1ZnjoHa9FNSbCxKKO2lxbc/Tbs7E3gVTnN8QCme n1ig9buIP2R0MJZY8fFoiOs6b0F5IU/w5d1AJV1KKSkbKD0XhtOwP+QmsQ5uzo1MaoQd z0yrxNkyH05nKUbEXVeXBi/Z9e1X1hTAGuoE8iGuZG0yAWavVgx44CCDAihDio0Qu5he WaeA== X-Forwarded-Encrypted: i=1; AJvYcCVlryiWUPp9h0rWQJlGeftRjP6HiERMiDyA1oSR9Q3TcBsNnW8UNlnGmI/S/ZzKqoDBjXibc4xDlrUmpsE=@vger.kernel.org X-Gm-Message-State: AOJu0Yw+p4ej5cNEVteDZhiQB7nicxHBBHftfGl34x+Lx0lsB7h7zrc6 f2WH1TWUAev5BftTm1wBJirW29P6Qn9z/h7ZFNfSn36KbCxb3jx2oU2de4nUJEZkSnUj47GI65g 2jLDUr8XcYOsCuiEXRA== X-Google-Smtp-Source: AGHT+IHp8tdxBCI9Q7uz6H+aiHlGXfLOJyM6qNf9CDG0PcOfLQK4TggfImr4/05l5Ruj7HVq1yX93CVhFQoj5Pw0 X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:13d:fb22:ac12:a84b]) (user=jthoughton job=sendgmr) by 2002:a25:9346:0:b0:e2b:cca2:69e1 with SMTP id 3f1490d57ef6-e2bcca26b2amr2712276.3.1729301386945; Fri, 18 Oct 2024 18:29:46 -0700 (PDT) Date: Sat, 19 Oct 2024 01:29:39 +0000 In-Reply-To: <20241019012940.3656292-1-jthoughton@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241019012940.3656292-1-jthoughton@google.com> X-Mailer: git-send-email 2.47.0.105.g07ac214952-goog Message-ID: <20241019012940.3656292-3-jthoughton@google.com> Subject: [PATCH 2/2] mm: multi-gen LRU: use {ptep,pmdp}_clear_young_notify() From: James Houghton To: Andrew Morton Cc: Sean Christopherson , Paolo Bonzini , David Matlack , David Rientjes , James Houghton , Oliver Upton , David Stevens , Yu Zhao , Wei Xu , Axel Rasmussen , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Yu Zhao When the MM_WALK capability is enabled, memory that is mostly accessed by a VM appears younger than it really is, therefore this memory will be less likely to be evicted. Therefore, the presence of a running VM can significantly increase swap-outs for non-VM memory, regressing the performance for the rest of the system. Fix this regression by always calling {ptep,pmdp}_clear_young_notify() whenever we clear the young bits on PMDs/PTEs. Fixes: bd74fdaea146 ("mm: multi-gen LRU: support page table walks") Reported-by: David Stevens Signed-off-by: Yu Zhao Signed-off-by: James Houghton --- include/linux/mmzone.h | 5 ++- mm/rmap.c | 9 ++--- mm/vmscan.c | 91 +++++++++++++++++++++++------------------- 3 files changed, 55 insertions(+), 50 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 691c635d8d1f..2e8c4307c728 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -557,7 +557,7 @@ struct lru_gen_memcg { =20 void lru_gen_init_pgdat(struct pglist_data *pgdat); void lru_gen_init_lruvec(struct lruvec *lruvec); -void lru_gen_look_around(struct page_vma_mapped_walk *pvmw); +bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw); =20 void lru_gen_init_memcg(struct mem_cgroup *memcg); void lru_gen_exit_memcg(struct mem_cgroup *memcg); @@ -576,8 +576,9 @@ static inline void lru_gen_init_lruvec(struct lruvec *l= ruvec) { } =20 -static inline void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) +static inline bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw) { + return false; } =20 static inline void lru_gen_init_memcg(struct mem_cgroup *memcg) diff --git a/mm/rmap.c b/mm/rmap.c index 2c561b1e52cc..4785a693857a 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -896,13 +896,10 @@ static bool folio_referenced_one(struct folio *folio, return false; } =20 - if (pvmw.pte) { - if (lru_gen_enabled() && - pte_young(ptep_get(pvmw.pte))) { - lru_gen_look_around(&pvmw); + if (lru_gen_enabled() && pvmw.pte) { + if (lru_gen_look_around(&pvmw)) referenced++; - } - + } else if (pvmw.pte) { if (ptep_clear_flush_young_notify(vma, address, pvmw.pte)) referenced++; diff --git a/mm/vmscan.c b/mm/vmscan.c index 60669f8bba46..29c098790b01 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -56,6 +56,7 @@ #include #include #include +#include =20 #include #include @@ -3293,7 +3294,8 @@ static bool get_next_vma(unsigned long mask, unsigned= long size, struct mm_walk return false; } =20 -static unsigned long get_pte_pfn(pte_t pte, struct vm_area_struct *vma, un= signed long addr) +static unsigned long get_pte_pfn(pte_t pte, struct vm_area_struct *vma, un= signed long addr, + struct pglist_data *pgdat) { unsigned long pfn =3D pte_pfn(pte); =20 @@ -3305,13 +3307,20 @@ static unsigned long get_pte_pfn(pte_t pte, struct = vm_area_struct *vma, unsigned if (WARN_ON_ONCE(pte_devmap(pte) || pte_special(pte))) return -1; =20 + if (!pte_young(pte) && !mm_has_notifiers(vma->vm_mm)) + return -1; + if (WARN_ON_ONCE(!pfn_valid(pfn))) return -1; =20 + if (pfn < pgdat->node_start_pfn || pfn >=3D pgdat_end_pfn(pgdat)) + return -1; + return pfn; } =20 -static unsigned long get_pmd_pfn(pmd_t pmd, struct vm_area_struct *vma, un= signed long addr) +static unsigned long get_pmd_pfn(pmd_t pmd, struct vm_area_struct *vma, un= signed long addr, + struct pglist_data *pgdat) { unsigned long pfn =3D pmd_pfn(pmd); =20 @@ -3323,9 +3332,15 @@ static unsigned long get_pmd_pfn(pmd_t pmd, struct v= m_area_struct *vma, unsigned if (WARN_ON_ONCE(pmd_devmap(pmd))) return -1; =20 + if (!pmd_young(pmd) && !mm_has_notifiers(vma->vm_mm)) + return -1; + if (WARN_ON_ONCE(!pfn_valid(pfn))) return -1; =20 + if (pfn < pgdat->node_start_pfn || pfn >=3D pgdat_end_pfn(pgdat)) + return -1; + return pfn; } =20 @@ -3334,10 +3349,6 @@ static struct folio *get_pfn_folio(unsigned long pfn= , struct mem_cgroup *memcg, { struct folio *folio; =20 - /* try to avoid unnecessary memory loads */ - if (pfn < pgdat->node_start_pfn || pfn >=3D pgdat_end_pfn(pgdat)) - return NULL; - folio =3D pfn_folio(pfn); if (folio_nid(folio) !=3D pgdat->node_id) return NULL; @@ -3400,20 +3411,16 @@ static bool walk_pte_range(pmd_t *pmd, unsigned lon= g start, unsigned long end, total++; walk->mm_stats[MM_LEAF_TOTAL]++; =20 - pfn =3D get_pte_pfn(ptent, args->vma, addr); + pfn =3D get_pte_pfn(ptent, args->vma, addr, pgdat); if (pfn =3D=3D -1) continue; =20 - if (!pte_young(ptent)) { - continue; - } - folio =3D get_pfn_folio(pfn, memcg, pgdat, walk->can_swap); if (!folio) continue; =20 - if (!ptep_test_and_clear_young(args->vma, addr, pte + i)) - VM_WARN_ON_ONCE(true); + if (!ptep_clear_young_notify(args->vma, addr, pte + i)) + continue; =20 young++; walk->mm_stats[MM_LEAF_YOUNG]++; @@ -3479,21 +3486,22 @@ static void walk_pmd_range_locked(pud_t *pud, unsig= ned long addr, struct vm_area /* don't round down the first address */ addr =3D i ? (*first & PMD_MASK) + i * PMD_SIZE : *first; =20 - pfn =3D get_pmd_pfn(pmd[i], vma, addr); - if (pfn =3D=3D -1) - goto next; - - if (!pmd_trans_huge(pmd[i])) { - if (!walk->force_scan && should_clear_pmd_young()) + if (pmd_present(pmd[i]) && !pmd_trans_huge(pmd[i])) { + if (!walk->force_scan && should_clear_pmd_young() && + !mm_has_notifiers(args->mm)) pmdp_test_and_clear_young(vma, addr, pmd + i); goto next; } =20 + pfn =3D get_pmd_pfn(pmd[i], vma, addr, pgdat); + if (pfn =3D=3D -1) + goto next; + folio =3D get_pfn_folio(pfn, memcg, pgdat, walk->can_swap); if (!folio) goto next; =20 - if (!pmdp_test_and_clear_young(vma, addr, pmd + i)) + if (!pmdp_clear_young_notify(vma, addr, pmd + i)) goto next; =20 walk->mm_stats[MM_LEAF_YOUNG]++; @@ -3551,24 +3559,18 @@ static void walk_pmd_range(pud_t *pud, unsigned lon= g start, unsigned long end, } =20 if (pmd_trans_huge(val)) { - unsigned long pfn =3D pmd_pfn(val); struct pglist_data *pgdat =3D lruvec_pgdat(walk->lruvec); + unsigned long pfn =3D get_pmd_pfn(val, vma, addr, pgdat); =20 walk->mm_stats[MM_LEAF_TOTAL]++; =20 - if (!pmd_young(val)) { - continue; - } - - /* try to avoid unnecessary memory loads */ - if (pfn < pgdat->node_start_pfn || pfn >=3D pgdat_end_pfn(pgdat)) - continue; - - walk_pmd_range_locked(pud, addr, vma, args, bitmap, &first); + if (pfn !=3D -1) + walk_pmd_range_locked(pud, addr, vma, args, bitmap, &first); continue; } =20 - if (!walk->force_scan && should_clear_pmd_young()) { + if (!walk->force_scan && should_clear_pmd_young() && + !mm_has_notifiers(args->mm)) { if (!pmd_young(val)) continue; =20 @@ -4042,13 +4044,13 @@ static void lru_gen_age_node(struct pglist_data *pg= dat, struct scan_control *sc) * the PTE table to the Bloom filter. This forms a feedback loop between t= he * eviction and the aging. */ -void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) +bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw) { int i; unsigned long start; unsigned long end; struct lru_gen_mm_walk *walk; - int young =3D 0; + int young =3D 1; pte_t *pte =3D pvmw->pte; unsigned long addr =3D pvmw->address; struct vm_area_struct *vma =3D pvmw->vma; @@ -4064,12 +4066,15 @@ void lru_gen_look_around(struct page_vma_mapped_wal= k *pvmw) lockdep_assert_held(pvmw->ptl); VM_WARN_ON_ONCE_FOLIO(folio_test_lru(folio), folio); =20 + if (!ptep_clear_young_notify(vma, addr, pte)) + return false; + if (spin_is_contended(pvmw->ptl)) - return; + return true; =20 /* exclude special VMAs containing anon pages from COW */ if (vma->vm_flags & VM_SPECIAL) - return; + return true; =20 /* avoid taking the LRU lock under the PTL when possible */ walk =3D current->reclaim_state ? current->reclaim_state->mm_walk : NULL; @@ -4077,6 +4082,9 @@ void lru_gen_look_around(struct page_vma_mapped_walk = *pvmw) start =3D max(addr & PMD_MASK, vma->vm_start); end =3D min(addr | ~PMD_MASK, vma->vm_end - 1) + 1; =20 + if (end - start =3D=3D PAGE_SIZE) + return true; + if (end - start > MIN_LRU_BATCH * PAGE_SIZE) { if (addr - start < MIN_LRU_BATCH * PAGE_SIZE / 2) end =3D start + MIN_LRU_BATCH * PAGE_SIZE; @@ -4090,7 +4098,7 @@ void lru_gen_look_around(struct page_vma_mapped_walk = *pvmw) =20 /* folio_update_gen() requires stable folio_memcg() */ if (!mem_cgroup_trylock_pages(memcg)) - return; + return true; =20 arch_enter_lazy_mmu_mode(); =20 @@ -4100,19 +4108,16 @@ void lru_gen_look_around(struct page_vma_mapped_wal= k *pvmw) unsigned long pfn; pte_t ptent =3D ptep_get(pte + i); =20 - pfn =3D get_pte_pfn(ptent, vma, addr); + pfn =3D get_pte_pfn(ptent, vma, addr, pgdat); if (pfn =3D=3D -1) continue; =20 - if (!pte_young(ptent)) - continue; - folio =3D get_pfn_folio(pfn, memcg, pgdat, can_swap); if (!folio) continue; =20 - if (!ptep_test_and_clear_young(vma, addr, pte + i)) - VM_WARN_ON_ONCE(true); + if (!ptep_clear_young_notify(vma, addr, pte + i)) + continue; =20 young++; =20 @@ -4144,6 +4149,8 @@ void lru_gen_look_around(struct page_vma_mapped_walk = *pvmw) /* feedback from rmap walkers to page table walkers */ if (mm_state && suitable_to_scan(i, young)) update_bloom_filter(mm_state, max_seq, pvmw->pmd); + + return true; } =20 /*************************************************************************= ***** --=20 2.47.0.105.g07ac214952-goog