From nobody Fri Dec 19 20:54:07 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 89AEC1940B3; Fri, 9 Aug 2024 11:49:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723204157; cv=none; b=slLCTM0NXl94p/GCA/ZOqwwgF51jbMM+pIoy8+J0Uw/nao4uqYBpYRwoUL2VlpGqZMQDtf7PTRiF0gSnfYCQbq784Ef0k/E8rdNR+tXm1VbLaefZwx3RRRxDjdMERQU3ZqZiW3DwJDYC8oso9sDiaSQOK5695vmk9/lyAHkbxDw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723204157; c=relaxed/simple; bh=SK6555nvVLN6liKsrnsOkA+XODJGhiDFiIQx9X5haBs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HhVIKy65bvsRpna2jsV+23c2DUqAW59ITvPPMkoSc8CJL+YQQWD4zSh1n3/18UZMLGuwz2phxB2aViEnI8Xs06Ro9gaMf2ysBPDiFaoe23xjDpwn2IFFzymshKdjuT1G2WRI97CZFZTXri/WeiuYDH1H72ztgJ2eSLFXq4ARbd8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.helo=mgamail.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=daypgCBf; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.helo=mgamail.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="daypgCBf" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1723204155; x=1754740155; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=SK6555nvVLN6liKsrnsOkA+XODJGhiDFiIQx9X5haBs=; b=daypgCBfoHTkW2GYEkxxvS+Msuq44yd486/g1S785+2zSdiV57zSG4L1 LPixAdIgm8tVDpEGlQEVQMhIqND/u9FGxA2XegTujRWdPAk40mFSDDCY0 aiw1Uh86LCdO+vxFqFKw9q6LDhQMx2N9sRLB1Ba/d7WAfPbiTCS9Rtb2j t+N1Wa3IrHRNoPp6mx8NP2t5riWo1mKnDHiGI7rmhOJcO3QoPuRvQoIOV n22pHfVPjjJG8kZ/Tid49KIrKBizWK7eE+zXqcYMkvbfs53/7QMErixev XL87HTHyCnjEK/2f0MM9/Sco38cSEEdbS9MJN2sLndjbSSGuX+bhpEyDV Q==; X-CSE-ConnectionGUID: i7/2CFVGTlOLu32oQpCdsA== X-CSE-MsgGUID: JqTTwyUnTj648fuEec23MQ== X-IronPort-AV: E=McAfee;i="6700,10204,11158"; a="24280321" X-IronPort-AV: E=Sophos;i="6.09,276,1716274800"; d="scan'208";a="24280321" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Aug 2024 04:49:10 -0700 X-CSE-ConnectionGUID: 7BLDRKQVTJG1D1Ux70Uajg== X-CSE-MsgGUID: qgriSKmHSL+s9l8ao6oQnA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,276,1716274800"; d="scan'208";a="57423600" Received: from black.fi.intel.com ([10.237.72.28]) by orviesa010.jf.intel.com with ESMTP; 09 Aug 2024 04:49:07 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 31F3919E; Fri, 09 Aug 2024 14:49:02 +0300 (EEST) From: "Kirill A. Shutemov" To: Andrew Morton , "Borislav Petkov (AMD)" , Mel Gorman , Vlastimil Babka Cc: Tom Lendacky , Mike Rapoport , "Matthew Wilcox (Oracle)" , David Hildenbrand , Johannes Weiner , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Jianxiong Gao , stable@vger.kernel.org Subject: [PATCHv2 1/8] mm: Fix endless reclaim on machines with unaccepted memory Date: Fri, 9 Aug 2024 14:48:47 +0300 Message-ID: <20240809114854.3745464-2-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240809114854.3745464-1-kirill.shutemov@linux.intel.com> References: <20240809114854.3745464-1-kirill.shutemov@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Unaccepted memory is considered unusable free memory, which is not counted as free on the zone watermark check. This causes get_page_from_freelist() to accept more memory to hit the high watermark, but it creates problems in the reclaim path. The reclaim path encounters a failed zone watermark check and attempts to reclaim memory. This is usually successful, but if there is little or no reclaimable memory, it can result in endless reclaim with little to no progress. This can occur early in the boot process, just after start of the init process when the only reclaimable memory is the page cache of the init executable and its libraries. Make unaccepted memory free from watermark check point of view. This way unaccepted memory will never be the trigger of memory reclaim. Accept more memory in the get_page_from_freelist() if needed. Signed-off-by: Kirill A. Shutemov Reported-by: Jianxiong Gao Acked-by: David Hildenbrand Fixes: dcdfdd40fa82 ("mm: Add support for unaccepted memory") Cc: stable@vger.kernel.org # v6.5+ Tested-by: Jianxiong Gao --- mm/page_alloc.c | 42 ++++++++++++++++++++---------------------- 1 file changed, 20 insertions(+), 22 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 5b5f99f4b20a..a9a89980f3f2 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -287,7 +287,7 @@ EXPORT_SYMBOL(nr_online_nodes); =20 static bool page_contains_unaccepted(struct page *page, unsigned int order= ); static void accept_page(struct page *page, unsigned int order); -static bool try_to_accept_memory(struct zone *zone, unsigned int order); +static bool cond_accept_memory(struct zone *zone, unsigned int order); static inline bool has_unaccepted_memory(void); static bool __free_unaccepted(struct page *page); =20 @@ -3108,9 +3108,6 @@ static inline long __zone_watermark_unusable_free(str= uct zone *z, if (!(alloc_flags & ALLOC_CMA)) unusable_free +=3D zone_page_state(z, NR_FREE_CMA_PAGES); #endif -#ifdef CONFIG_UNACCEPTED_MEMORY - unusable_free +=3D zone_page_state(z, NR_UNACCEPTED); -#endif =20 return unusable_free; } @@ -3404,6 +3401,8 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int o= rder, int alloc_flags, } } =20 + cond_accept_memory(zone, order); + /* * Detect whether the number of free pages is below high * watermark. If so, we will decrease pcp->high and free @@ -3429,10 +3428,8 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int = order, int alloc_flags, gfp_mask)) { int ret; =20 - if (has_unaccepted_memory()) { - if (try_to_accept_memory(zone, order)) - goto try_this_zone; - } + if (cond_accept_memory(zone, order)) + goto try_this_zone; =20 #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT /* @@ -3486,10 +3483,8 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int = order, int alloc_flags, =20 return page; } else { - if (has_unaccepted_memory()) { - if (try_to_accept_memory(zone, order)) - goto try_this_zone; - } + if (cond_accept_memory(zone, order)) + goto try_this_zone; =20 #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT /* Try again if zone has deferred pages */ @@ -6988,9 +6983,6 @@ static bool try_to_accept_memory_one(struct zone *zon= e) struct page *page; bool last; =20 - if (list_empty(&zone->unaccepted_pages)) - return false; - spin_lock_irqsave(&zone->lock, flags); page =3D list_first_entry_or_null(&zone->unaccepted_pages, struct page, lru); @@ -7016,23 +7008,29 @@ static bool try_to_accept_memory_one(struct zone *z= one) return true; } =20 -static bool try_to_accept_memory(struct zone *zone, unsigned int order) +static bool cond_accept_memory(struct zone *zone, unsigned int order) { long to_accept; - int ret =3D false; + bool ret =3D false; + + if (!has_unaccepted_memory()) + return false; + + if (list_empty(&zone->unaccepted_pages)) + return false; =20 /* How much to accept to get to high watermark? */ to_accept =3D high_wmark_pages(zone) - (zone_page_state(zone, NR_FREE_PAGES) - - __zone_watermark_unusable_free(zone, order, 0)); + __zone_watermark_unusable_free(zone, order, 0) - + zone_page_state(zone, NR_UNACCEPTED)); =20 - /* Accept at least one page */ - do { + while (to_accept > 0) { if (!try_to_accept_memory_one(zone)) break; ret =3D true; to_accept -=3D MAX_ORDER_NR_PAGES; - } while (to_accept > 0); + } =20 return ret; } @@ -7075,7 +7073,7 @@ static void accept_page(struct page *page, unsigned i= nt order) { } =20 -static bool try_to_accept_memory(struct zone *zone, unsigned int order) +static bool cond_accept_memory(struct zone *zone, unsigned int order) { return false; } --=20 2.43.0 From nobody Fri Dec 19 20:54:07 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 79A1819308F for ; Fri, 9 Aug 2024 11:49:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723204151; cv=none; b=GDpME3PeiEAmlVfY88W4h0pWVQaVDmpcWPJaDsc7z9yMGb1nQDRsxnCB6/LDBXX9MCsnBt6t3diW+fcdEiY//ipoq9F8LGEGFuwGCtCtCNrYrhX4qHWq5REV2ju6ePBmye2vt1Eph4rpHHxoHfI51oQpUCTZc7co+Sca5sQH/EE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723204151; c=relaxed/simple; bh=8uQZF5Uhz2ovztR/I7NMgOMJfpN3cl6gEDHYcVKTSDE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lHuvAvTihqWmN2PrzAJtIXnfPCtpHlkyatVM/W4MUJNF/gZkemlCga/w86JjACbDNBALX1iyL/O8HDTpUuyKeskDOYD+kC8TasA/2pIW3ySUt+JwtpHC4EDyWC07p20e/iQNC44fMaygjzBY0XXjlodWKryOAzSsq/5byl8julc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.helo=mgamail.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=RSpFxGTY; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.helo=mgamail.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="RSpFxGTY" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1723204151; x=1754740151; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8uQZF5Uhz2ovztR/I7NMgOMJfpN3cl6gEDHYcVKTSDE=; b=RSpFxGTYaxgsG+7YdWeCDe27xKaoLT7lDcDSAY+CKiJXJjYoCp3b3dm+ AThQ+sf0giNNI1gp43ju8snnXo+tNPSuajYmK7xr57aXoH61u5K+AWOrG 2er0CZ9lF9yHsXQHNLA3jy5BNdhKNdHCG4HmirjknvC/GY8VLRwT94HV2 meHYGat0ACEuUrknDC1yhWSmblnnbGlhlIoGyccDvqtf+U62h1lLsYcTy nEJD1TosuGPOq0iaJAal21n9XKSWXHYWIaIBeDRATaIC4QzrofDg22OZ5 siq55UaZVlWhavT/4iVRio/jtt52SLhPfNzSe0SNNTYra+vo8CpENKz8F w==; X-CSE-ConnectionGUID: Bfa/Nx3XTMG2mUtelnW/vg== X-CSE-MsgGUID: kkeMamHXTz+30V/TzCB5Lw== X-IronPort-AV: E=McAfee;i="6700,10204,11158"; a="24280286" X-IronPort-AV: E=Sophos;i="6.09,276,1716274800"; d="scan'208";a="24280286" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Aug 2024 04:49:10 -0700 X-CSE-ConnectionGUID: 8I4v0cgrQSGm09kCFwGCrg== X-CSE-MsgGUID: 44H2KFQgQ/C/ttA/2Zxb9w== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,276,1716274800"; d="scan'208";a="57423598" Received: from black.fi.intel.com ([10.237.72.28]) by orviesa010.jf.intel.com with ESMTP; 09 Aug 2024 04:49:07 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 3C4E3597; Fri, 09 Aug 2024 14:49:02 +0300 (EEST) From: "Kirill A. Shutemov" To: Andrew Morton , "Borislav Petkov (AMD)" , Mel Gorman , Vlastimil Babka Cc: Tom Lendacky , Mike Rapoport , "Matthew Wilcox (Oracle)" , David Hildenbrand , Johannes Weiner , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv2 2/8] mm: Reduce deferred struct page init ifdeffery Date: Fri, 9 Aug 2024 14:48:48 +0300 Message-ID: <20240809114854.3745464-3-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240809114854.3745464-1-kirill.shutemov@linux.intel.com> References: <20240809114854.3745464-1-kirill.shutemov@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add dummy _deferred_grow_zone() for !DEFERRED_STRUCT_PAGE_INIT and remove #ifdefs in two places. No functional changes. Signed-off-by: Kirill A. Shutemov Suggested-by: David Hildenbrand --- mm/page_alloc.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a9a89980f3f2..f7bb885aab07 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -322,6 +322,11 @@ static inline bool deferred_pages_enabled(void) { return false; } + +static inline bool _deferred_grow_zone(struct zone *zone, unsigned int ord= er) +{ + return false; +} #endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ =20 /* Return a pointer to the bitmap storing bits affecting a block of pages = */ @@ -3431,7 +3436,6 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int o= rder, int alloc_flags, if (cond_accept_memory(zone, order)) goto try_this_zone; =20 -#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT /* * Watermark failed for this zone, but see if we can * grow this zone if it contains deferred pages. @@ -3440,7 +3444,6 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int o= rder, int alloc_flags, if (_deferred_grow_zone(zone, order)) goto try_this_zone; } -#endif /* Checked here to keep the fast path fast */ BUILD_BUG_ON(ALLOC_NO_WATERMARKS < NR_WMARK); if (alloc_flags & ALLOC_NO_WATERMARKS) @@ -3486,13 +3489,11 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int= order, int alloc_flags, if (cond_accept_memory(zone, order)) goto try_this_zone; =20 -#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT /* Try again if zone has deferred pages */ if (deferred_pages_enabled()) { if (_deferred_grow_zone(zone, order)) goto try_this_zone; } -#endif } } =20 --=20 2.43.0 From nobody Fri Dec 19 20:54:07 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0176619309D for ; Fri, 9 Aug 2024 11:49:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723204153; cv=none; b=jYpAYTBWU17VvwyjTgmAzVxAtGqIjsFejj4L/wnX+Mb9GXB/IuMgFBF7HCs4e3+SLRa8ZMBHW7FAEXAvk6cK15FteY0LJsTDbvR7FMqf/oxHRpVVBO1m6Nko8cJK09HuYTpreEYCrREUClGdAyJ7kTazp8ZBEACSKC37ciI3dqI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723204153; c=relaxed/simple; bh=0wj6Qv7exseEtKqqNf9edLNjDoFKcqwOubwV0S+NfTY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WdP01JIORky9AudrvVR1SX38b0ijeArgMm5YhbfKPjhRrcLLQ/XaqWKvZFlhi3JlXNKkOBXzGj4fDWYvr6IvDqfo3IV3DwQDkbz5yt8iQlT5ZIaBCi6ees6FNLt/YI5Glrn+21tFGk+fGibtYm4znNY799ZhR+PbntnRWwgYPeo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.helo=mgamail.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=loxPrx2s; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.helo=mgamail.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="loxPrx2s" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1723204152; x=1754740152; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=0wj6Qv7exseEtKqqNf9edLNjDoFKcqwOubwV0S+NfTY=; b=loxPrx2soxdki14sx6bpfJMb9sbq/c4UymUu8fhpSJCMbMD0WUhygc2p xDOau+qsAhPhqkW56AxdpHpUwO1eTI4TdX4EMmCVEiBnDFxbt2vmsu47v sgS2vLYzm19cQANZSU4FB4A/9V4h7qVk9e1C7BUpYlGSBsgkch8vMP5QX uulb1oUruBsf707yWhjx84TfRhNssndIHltR89S576/IJpzFALC2g2j3v AYq/qB3Fu63O5rbAhvGx0ANOLGCZqUah3MyKUy/XnPMk7dfzMjxaJ39v1 gRPzprYVfR8bcOIwXKBPJSTZiwBURB7eh1dr78JcD9/DdzlJTVkFnJPC6 Q==; X-CSE-ConnectionGUID: qQTMyzRISNmuG8t/6yN2Wg== X-CSE-MsgGUID: VuEuG2U2SNqhTmk8WuPQOw== X-IronPort-AV: E=McAfee;i="6700,10204,11158"; a="24280309" X-IronPort-AV: E=Sophos;i="6.09,276,1716274800"; d="scan'208";a="24280309" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Aug 2024 04:49:10 -0700 X-CSE-ConnectionGUID: ZY8VDZEHSQ+33d93qhtE2Q== X-CSE-MsgGUID: kj1qNfb1Q1qGIzf4Di2OXw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,276,1716274800"; d="scan'208";a="57423601" Received: from black.fi.intel.com ([10.237.72.28]) by orviesa010.jf.intel.com with ESMTP; 09 Aug 2024 04:49:07 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 48B0097E; Fri, 09 Aug 2024 14:49:02 +0300 (EEST) From: "Kirill A. Shutemov" To: Andrew Morton , "Borislav Petkov (AMD)" , Mel Gorman , Vlastimil Babka Cc: Tom Lendacky , Mike Rapoport , "Matthew Wilcox (Oracle)" , David Hildenbrand , Johannes Weiner , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv2 3/8] mm: Accept memory in __alloc_pages_bulk(). Date: Fri, 9 Aug 2024 14:48:49 +0300 Message-ID: <20240809114854.3745464-4-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240809114854.3745464-1-kirill.shutemov@linux.intel.com> References: <20240809114854.3745464-1-kirill.shutemov@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, the kernel only accepts memory in get_page_from_freelist(), but there is another path that directly takes pages from free lists - __alloc_page_bulk(). This function can consume all accepted memory and will resort to __alloc_pages_noprof() if necessary. Conditionally accepted in __alloc_pages_bulk(). The same issue may arise due to deferred page initialization. Kick the deferred initialization machinery before abandoning the zone, as the kernel does in get_page_from_freelist(). Signed-off-by: Kirill A. Shutemov Acked-by: David Hildenbrand --- mm/page_alloc.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index f7bb885aab07..ed62ecd6775f 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4613,12 +4613,23 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, in= t preferred_nid, goto failed; } =20 + cond_accept_memory(zone, 0); +retry_this_zone: mark =3D wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK) + nr_pages; if (zone_watermark_fast(zone, 0, mark, zonelist_zone_idx(ac.preferred_zoneref), alloc_flags, gfp)) { break; } + + if (cond_accept_memory(zone, 0)) + goto retry_this_zone; + + /* Try again if zone has deferred pages */ + if (deferred_pages_enabled()) { + if (_deferred_grow_zone(zone, 0)) + goto retry_this_zone; + } } =20 /* --=20 2.43.0 From nobody Fri Dec 19 20:54:07 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0676B19309E for ; Fri, 9 Aug 2024 11:49:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723204153; cv=none; b=U8aO3wyO5kmGb9TLAvLEzddSyJI7Dc9TEocVg+4LJnpOVNOZ2BRbbCySvtZbqe63FLQMjMAKHPyhtLqY/ncdM1kzWwoi//Er4mVo3Z3iniYB923b6FioJa2ROhgFlYMR5/MVnmIoijvobCB91bt50GyhCUGoocDo6RqvAoyOiTk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723204153; c=relaxed/simple; bh=tiHFKee3PrdQg/q59/O8kANn06DyhKlIl9RN3/Z3JdM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dn+t4mNfqjFEWy1fA0ebUykdmyJ4GxlMIO/TIeCmfyIM4XZWDbJn7FxXwG6cgVqPkX+pChR4Udbzs5b8JSEihgluyAB+sHCG6v7j2He4IvLysEfGt0w+fVRppCplMmD7Mvca5mZgUtD+fXhg5e7/12Lb5m5n3JU4p+3taK8q768= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.helo=mgamail.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=e6TSFpMw; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.helo=mgamail.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="e6TSFpMw" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1723204152; x=1754740152; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=tiHFKee3PrdQg/q59/O8kANn06DyhKlIl9RN3/Z3JdM=; b=e6TSFpMws82C+DaYXOHM3FTw4Y+iFNXte4BNuBrbyM6UP7JHrrwOgSPT jCtMF6gKhlP7E3AWjDnPaNyAKcRhAcpiNfqaqzRBxPaEWqu4VSjOTTyyN 5INcU3pr50r/cW/kE08yco/+P6S48bmtBwczjJOpgDDniiaLKUi1LZtCr 8I1Iy0vihvEqhtU7iXBhKGKIfOAuoKsNUiODsxV5WVrXSVJBcd0JJC+R6 7dLV2MEG2iE8Bw1xSJB3G5zGhtZppN0NW9AEPlGGjpG4z+VTZJ+qTBB5k +crLah4j95XXB+tTV1jUDsqvmO52YzlSJfz7Goj55O8XUYvCLYvIYv1Gd Q==; X-CSE-ConnectionGUID: lmrNCFCJQ4m776HxDGmIYA== X-CSE-MsgGUID: U8eNsVTCTNCcekPmARJtsQ== X-IronPort-AV: E=McAfee;i="6700,10204,11158"; a="24280297" X-IronPort-AV: E=Sophos;i="6.09,276,1716274800"; d="scan'208";a="24280297" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Aug 2024 04:49:10 -0700 X-CSE-ConnectionGUID: cW0eGtWmR0KLF4LZPKS19A== X-CSE-MsgGUID: ti0C+JuCS1mYpXpUXFIYhg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,276,1716274800"; d="scan'208";a="57423599" Received: from black.fi.intel.com ([10.237.72.28]) by orviesa010.jf.intel.com with ESMTP; 09 Aug 2024 04:49:07 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 517609C0; Fri, 09 Aug 2024 14:49:02 +0300 (EEST) From: "Kirill A. Shutemov" To: Andrew Morton , "Borislav Petkov (AMD)" , Mel Gorman , Vlastimil Babka Cc: Tom Lendacky , Mike Rapoport , "Matthew Wilcox (Oracle)" , David Hildenbrand , Johannes Weiner , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv2 4/8] mm: Introduce PageUnaccepted() page type Date: Fri, 9 Aug 2024 14:48:50 +0300 Message-ID: <20240809114854.3745464-5-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240809114854.3745464-1-kirill.shutemov@linux.intel.com> References: <20240809114854.3745464-1-kirill.shutemov@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The new page type allows physical memory scanners to detect unaccepted memory and handle it accordingly. The page type is serialized with zone lock. Signed-off-by: Kirill A. Shutemov Acked-by: David Hildenbrand --- include/linux/page-flags.h | 8 ++++++++ mm/page_alloc.c | 2 ++ 2 files changed, 10 insertions(+) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index a0a29bd092f8..17175a328e6b 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -939,6 +939,7 @@ enum pagetype { PG_hugetlb =3D 0x04000000, PG_slab =3D 0x02000000, PG_zsmalloc =3D 0x01000000, + PG_unaccepted =3D 0x00800000, =20 PAGE_TYPE_BASE =3D 0x80000000, =20 @@ -1072,6 +1073,13 @@ FOLIO_TEST_FLAG_FALSE(hugetlb) =20 PAGE_TYPE_OPS(Zsmalloc, zsmalloc, zsmalloc) =20 +/* + * Mark pages that has to be accepted before touched for the first time. + * + * Serialized with zone lock. + */ +PAGE_TYPE_OPS(Unaccepted, unaccepted, unaccepted) + /** * PageHuge - Determine if the page belongs to hugetlbfs * @page: The page to test. diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ed62ecd6775f..f4368c2fa395 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -7008,6 +7008,7 @@ static bool try_to_accept_memory_one(struct zone *zon= e) =20 account_freepages(zone, -MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE); __mod_zone_page_state(zone, NR_UNACCEPTED, -MAX_ORDER_NR_PAGES); + __ClearPageUnaccepted(page); spin_unlock_irqrestore(&zone->lock, flags); =20 accept_page(page, MAX_PAGE_ORDER); @@ -7066,6 +7067,7 @@ static bool __free_unaccepted(struct page *page) list_add_tail(&page->lru, &zone->unaccepted_pages); account_freepages(zone, MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE); __mod_zone_page_state(zone, NR_UNACCEPTED, MAX_ORDER_NR_PAGES); + __SetPageUnaccepted(page); spin_unlock_irqrestore(&zone->lock, flags); =20 if (first) --=20 2.43.0 From nobody Fri Dec 19 20:54:07 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 85C02193091 for ; Fri, 9 Aug 2024 11:49:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723204152; cv=none; b=qXixcyV4MBQd+9jZKV30sbbmQCH703m/eZsx1adzVxXQW4mon3IedlA247/HqwnAlOFza09GlHyf2HATUVuoIUZAo3QgrRl8nXvOPamPmBaBkEJHWlxdmjP6L2jva9cOdoO7WKKCzhwtLpCXRi0yEWQnKi6MZDmKqZEJqP4w1XE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723204152; c=relaxed/simple; bh=acTVWN+uIbt5wIMQ6n+U5b8XonpliA5KJhBSmKOHvF8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=P7l92w3fOTV2Shx83CHVEOU0DmtXsupoPWl4SK1HIFP7NBUutq3eh/woE2K27r+oRrX6odl427hQpVHzsW47td+UZG9PW+Zg9Fb2/ueiHBbVDPz6EAIqsgcHBX+FqcGfHBHA8e4Xaek2qk5H3hcZCIwBDyu7rPjkoXecvMIOA1Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.helo=mgamail.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=XDRE0yQF; arc=none smtp.client-ip=192.198.163.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.helo=mgamail.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="XDRE0yQF" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1723204150; x=1754740150; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=acTVWN+uIbt5wIMQ6n+U5b8XonpliA5KJhBSmKOHvF8=; b=XDRE0yQFBTR+yr7R1grSH/sgQhUza7ToFnefO2s82NzcFKr0z23hKLp0 3UaQ+6wYv0JoxMx+FMB1AHJ8CxDL/lrBi28V8uOehxol4Q5molaoKXBL7 wpRx7HZS23+gJO0UNUEfo40G75yXiDtJ0Tj/q4Ym/WewJw9qtNmTxKpRW SkHTh7vA06c04BZxy7+7DimP/EiHmGkmeKFjIaPnVh7rxWbFZ5vqtqQ4Z IkhzE53Ec7Vtgot+FQ6NeObVGIwvSAeYOejYNyXj+5197EcNW1I38U4Cj 9VhHnZYnT9ZKNPyDH6vuKWmyy5EVuNPxE0ERGbRMOkEL9jpEb2rxnuKmT w==; X-CSE-ConnectionGUID: 6lN8DVTOSgWgdc/b5H7hhA== X-CSE-MsgGUID: Y8q+u9pESjOqFeGgYG3FSA== X-IronPort-AV: E=McAfee;i="6700,10204,11158"; a="31999753" X-IronPort-AV: E=Sophos;i="6.09,276,1716274800"; d="scan'208";a="31999753" Received: from orviesa007.jf.intel.com ([10.64.159.147]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Aug 2024 04:49:10 -0700 X-CSE-ConnectionGUID: HrJbWXijSgeDGakcMm7nDg== X-CSE-MsgGUID: vzukHdihTHeKv189XVMudw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,276,1716274800"; d="scan'208";a="58108832" Received: from black.fi.intel.com ([10.237.72.28]) by orviesa007.jf.intel.com with ESMTP; 09 Aug 2024 04:49:07 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 5D9029E0; Fri, 09 Aug 2024 14:49:02 +0300 (EEST) From: "Kirill A. Shutemov" To: Andrew Morton , "Borislav Petkov (AMD)" , Mel Gorman , Vlastimil Babka Cc: Tom Lendacky , Mike Rapoport , "Matthew Wilcox (Oracle)" , David Hildenbrand , Johannes Weiner , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv2 5/8] mm: Rework accept memory helpers Date: Fri, 9 Aug 2024 14:48:51 +0300 Message-ID: <20240809114854.3745464-6-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240809114854.3745464-1-kirill.shutemov@linux.intel.com> References: <20240809114854.3745464-1-kirill.shutemov@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Make accept_memory() and range_contains_unaccepted_memory() take 'start' and 'size' arguments instead of 'start' and 'end'. Remove accept_page(), replacing it with direct calls to accept_memory(). The accept_page() name is going to be used for a different function. Signed-off-by: Kirill A. Shutemov Suggested-by: David Hildenbrand --- arch/x86/boot/compressed/misc.c | 2 +- arch/x86/boot/compressed/misc.h | 2 +- drivers/firmware/efi/libstub/efistub.h | 2 +- .../firmware/efi/libstub/unaccepted_memory.c | 3 ++- drivers/firmware/efi/unaccepted_memory.c | 18 ++++++++++-------- include/linux/mm.h | 12 +++++------- mm/memblock.c | 2 +- mm/mm_init.c | 2 +- mm/page_alloc.c | 19 +++---------------- tools/testing/memblock/internal.h | 2 +- 10 files changed, 26 insertions(+), 38 deletions(-) diff --git a/arch/x86/boot/compressed/misc.c b/arch/x86/boot/compressed/mis= c.c index 944454306ef4..04a35b2c26e9 100644 --- a/arch/x86/boot/compressed/misc.c +++ b/arch/x86/boot/compressed/misc.c @@ -511,7 +511,7 @@ asmlinkage __visible void *extract_kernel(void *rmode, = unsigned char *output) =20 if (init_unaccepted_memory()) { debug_putstr("Accepting memory... "); - accept_memory(__pa(output), __pa(output) + needed_size); + accept_memory(__pa(output), needed_size); } =20 entry_offset =3D decompress_kernel(output, virt_addr, error); diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/mis= c.h index b353a7be380c..dd8d1a85f671 100644 --- a/arch/x86/boot/compressed/misc.h +++ b/arch/x86/boot/compressed/misc.h @@ -256,6 +256,6 @@ static inline bool init_unaccepted_memory(void) { retur= n false; } =20 /* Defined in EFI stub */ extern struct efi_unaccepted_memory *unaccepted_table; -void accept_memory(phys_addr_t start, phys_addr_t end); +void accept_memory(phys_addr_t start, unsigned long size); =20 #endif /* BOOT_COMPRESSED_MISC_H */ diff --git a/drivers/firmware/efi/libstub/efistub.h b/drivers/firmware/efi/= libstub/efistub.h index d33ccbc4a2c6..685098f9626f 100644 --- a/drivers/firmware/efi/libstub/efistub.h +++ b/drivers/firmware/efi/libstub/efistub.h @@ -1229,7 +1229,7 @@ efi_zboot_entry(efi_handle_t handle, efi_system_table= _t *systab); efi_status_t allocate_unaccepted_bitmap(__u32 nr_desc, struct efi_boot_memmap *map); void process_unaccepted_memory(u64 start, u64 end); -void accept_memory(phys_addr_t start, phys_addr_t end); +void accept_memory(phys_addr_t start, unsigned long size); void arch_accept_memory(phys_addr_t start, phys_addr_t end); =20 #endif diff --git a/drivers/firmware/efi/libstub/unaccepted_memory.c b/drivers/fir= mware/efi/libstub/unaccepted_memory.c index c295ea3a6efc..757dbe734a47 100644 --- a/drivers/firmware/efi/libstub/unaccepted_memory.c +++ b/drivers/firmware/efi/libstub/unaccepted_memory.c @@ -177,9 +177,10 @@ void process_unaccepted_memory(u64 start, u64 end) start / unit_size, (end - start) / unit_size); } =20 -void accept_memory(phys_addr_t start, phys_addr_t end) +void accept_memory(phys_addr_t start, unsigned long size) { unsigned long range_start, range_end; + phys_addr_t end =3D start + size; unsigned long bitmap_size; u64 unit_size; =20 diff --git a/drivers/firmware/efi/unaccepted_memory.c b/drivers/firmware/ef= i/unaccepted_memory.c index 50f6503fe49f..c2c067eff634 100644 --- a/drivers/firmware/efi/unaccepted_memory.c +++ b/drivers/firmware/efi/unaccepted_memory.c @@ -30,11 +30,12 @@ static LIST_HEAD(accepting_list); * - memory that is below phys_base; * - memory that is above the memory that addressable by the bitmap; */ -void accept_memory(phys_addr_t start, phys_addr_t end) +void accept_memory(phys_addr_t start, unsigned long size) { struct efi_unaccepted_memory *unaccepted; unsigned long range_start, range_end; struct accept_range range, *entry; + phys_addr_t end =3D start + size; unsigned long flags; u64 unit_size; =20 @@ -74,13 +75,13 @@ void accept_memory(phys_addr_t start, phys_addr_t end) * "guard" page is accepted in addition to the memory that needs to be * used: * - * 1. Implicitly extend the range_contains_unaccepted_memory(start, end) - * checks up to end+unit_size if 'end' is aligned on a unit_size - * boundary. + * 1. Implicitly extend the range_contains_unaccepted_memory(start, size) + * checks up to the next unit_size if 'start+size' is aligned on a + * unit_size boundary. * - * 2. Implicitly extend accept_memory(start, end) to end+unit_size if - * 'end' is aligned on a unit_size boundary. (immediately following - * this comment) + * 2. Implicitly extend accept_memory(start, size) to the next unit_size + * if 'size+end' is aligned on a unit_size boundary. (immediately + * following this comment) */ if (!(end % unit_size)) end +=3D unit_size; @@ -156,9 +157,10 @@ void accept_memory(phys_addr_t start, phys_addr_t end) spin_unlock_irqrestore(&unaccepted_memory_lock, flags); } =20 -bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end) +bool range_contains_unaccepted_memory(phys_addr_t start, unsigned long siz= e) { struct efi_unaccepted_memory *unaccepted; + phys_addr_t end =3D start + size; unsigned long flags; bool ret =3D false; u64 unit_size; diff --git a/include/linux/mm.h b/include/linux/mm.h index ee8cea73d415..b1eed30fdc06 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4062,18 +4062,18 @@ madvise_set_anon_name(struct mm_struct *mm, unsigne= d long start, =20 #ifdef CONFIG_UNACCEPTED_MEMORY =20 -bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end); -void accept_memory(phys_addr_t start, phys_addr_t end); +bool range_contains_unaccepted_memory(phys_addr_t start, unsigned long siz= e); +void accept_memory(phys_addr_t start, unsigned long size); =20 #else =20 static inline bool range_contains_unaccepted_memory(phys_addr_t start, - phys_addr_t end) + unsigned long size) { return false; } =20 -static inline void accept_memory(phys_addr_t start, phys_addr_t end) +static inline void accept_memory(phys_addr_t start, unsigned long size) { } =20 @@ -4081,9 +4081,7 @@ static inline void accept_memory(phys_addr_t start, p= hys_addr_t end) =20 static inline bool pfn_is_unaccepted_memory(unsigned long pfn) { - phys_addr_t paddr =3D pfn << PAGE_SHIFT; - - return range_contains_unaccepted_memory(paddr, paddr + PAGE_SIZE); + return range_contains_unaccepted_memory(pfn << PAGE_SHIFT, PAGE_SIZE); } =20 void vma_pgtable_walk_begin(struct vm_area_struct *vma); diff --git a/mm/memblock.c b/mm/memblock.c index 3b9dc2d89b8a..0a77a748a8eb 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -1500,7 +1500,7 @@ phys_addr_t __init memblock_alloc_range_nid(phys_addr= _t size, * * Accept the memory of the allocated buffer. */ - accept_memory(found, found + size); + accept_memory(found, size); =20 return found; } diff --git a/mm/mm_init.c b/mm/mm_init.c index 66e596c0129b..7a1a05cb048e 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1933,7 +1933,7 @@ static void __init deferred_free_pages(unsigned long = pfn, } =20 /* Accept chunks smaller than MAX_PAGE_ORDER upfront */ - accept_memory(PFN_PHYS(pfn), PFN_PHYS(pfn + nr_pages)); + accept_memory(PFN_PHYS(pfn), nr_pages * PAGE_SIZE); =20 for (i =3D 0; i < nr_pages; i++, page++, pfn++) { if (pageblock_aligned(pfn)) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index f4368c2fa395..e7e304d9e739 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -286,7 +286,6 @@ EXPORT_SYMBOL(nr_online_nodes); #endif =20 static bool page_contains_unaccepted(struct page *page, unsigned int order= ); -static void accept_page(struct page *page, unsigned int order); static bool cond_accept_memory(struct zone *zone, unsigned int order); static inline bool has_unaccepted_memory(void); static bool __free_unaccepted(struct page *page); @@ -1268,7 +1267,7 @@ void __meminit __free_pages_core(struct page *page, u= nsigned int order, if (order =3D=3D MAX_PAGE_ORDER && __free_unaccepted(page)) return; =20 - accept_page(page, order); + accept_memory(page_to_phys(page), PAGE_SIZE << order); } =20 /* @@ -6977,16 +6976,8 @@ early_param("accept_memory", accept_memory_parse); static bool page_contains_unaccepted(struct page *page, unsigned int order) { phys_addr_t start =3D page_to_phys(page); - phys_addr_t end =3D start + (PAGE_SIZE << order); =20 - return range_contains_unaccepted_memory(start, end); -} - -static void accept_page(struct page *page, unsigned int order) -{ - phys_addr_t start =3D page_to_phys(page); - - accept_memory(start, start + (PAGE_SIZE << order)); + return range_contains_unaccepted_memory(start, PAGE_SIZE << order); } =20 static bool try_to_accept_memory_one(struct zone *zone) @@ -7011,7 +7002,7 @@ static bool try_to_accept_memory_one(struct zone *zon= e) __ClearPageUnaccepted(page); spin_unlock_irqrestore(&zone->lock, flags); =20 - accept_page(page, MAX_PAGE_ORDER); + accept_memory(page_to_phys(page), PAGE_SIZE << MAX_PAGE_ORDER); =20 __free_pages_ok(page, MAX_PAGE_ORDER, FPI_TO_TAIL); =20 @@ -7083,10 +7074,6 @@ static bool page_contains_unaccepted(struct page *pa= ge, unsigned int order) return false; } =20 -static void accept_page(struct page *page, unsigned int order) -{ -} - static bool cond_accept_memory(struct zone *zone, unsigned int order) { return false; diff --git a/tools/testing/memblock/internal.h b/tools/testing/memblock/int= ernal.h index f6c6e5474c3a..1cf82acb2a3e 100644 --- a/tools/testing/memblock/internal.h +++ b/tools/testing/memblock/internal.h @@ -20,7 +20,7 @@ void memblock_free_pages(struct page *page, unsigned long= pfn, { } =20 -static inline void accept_memory(phys_addr_t start, phys_addr_t end) +static inline void accept_memory(phys_addr_t start, unsigned long size) { } =20 --=20 2.43.0 From nobody Fri Dec 19 20:54:07 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 946D0192B8A for ; Fri, 9 Aug 2024 11:49:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723204156; cv=none; b=X6yaG8lkZOyrH4scs1TyDh/I7IlUyIpE3IxH2WVb6P7nQKQE6/SoXjztMiHKNYj01891mtsGcDbrjZmATjX5jubNXD7oCYlj73H3O+/g5jHLfkV31zXLihE2Hv3rnBkPVTEIxoX0pExAMQP7jWoq/YNNbSia0H9sv5H9DZMANr8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723204156; c=relaxed/simple; bh=SzZoS9g7jt+luDGgY5b8GrGZZk7RG0DnODBwUGbdsd4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=T7bPC7z/ZhQIdPVpUzwkil/cVSgWCHssPGPzYcs2423a52j8actNwxjGzAnwj/YX8bgp9yEQAtkXd2IFWKGhA6khBbn1y/Y7Hw5FYnxWOMlqlVrd5wtPMQYlTOyMgvnBKH6Pk0VZYaQBSYq6j4Ppa2ZPKgb/2nSgBtlkWOb63U8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.helo=mgamail.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=XpY5q/lv; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.helo=mgamail.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="XpY5q/lv" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1723204155; x=1754740155; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=SzZoS9g7jt+luDGgY5b8GrGZZk7RG0DnODBwUGbdsd4=; b=XpY5q/lvLIfXTa6Wjfk0fuBdqvwY/gJHiwVKToKC7D9oHS0WuvvBN1Vc q9MvWq719FtxPnM8FliYTjTB/UeaEkdr5HBdnzPIbJvfHcWIkUTQup6wq zevXzTHRAKx7edtdR8iXP95x35U/OmDc/i33quBCAAI5JyDR6g/iujkIH dk/mzpaZpzqyySN+OIpFuVpEfaTuFqW4GKBAXuMaEF4AdnheAe71ENXdg qp4m3zjVOomXj5wyWAZXjLlOQA0wiP9jTFqozd2yKCY4bbyVKCnaN71zX Bv12LdJx/zl4M8V5rBNqunQJjps5tnXzNR75hbPwYFKH065dJQm2mDJSe w==; X-CSE-ConnectionGUID: 1BxHH9LhQv+r+4stuSyOtw== X-CSE-MsgGUID: P1JkqN6yQACHvMeTZg7Q/g== X-IronPort-AV: E=McAfee;i="6700,10204,11158"; a="24280336" X-IronPort-AV: E=Sophos;i="6.09,276,1716274800"; d="scan'208";a="24280336" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Aug 2024 04:49:12 -0700 X-CSE-ConnectionGUID: qoa/OEsTQqCUYKdEPPPs8A== X-CSE-MsgGUID: RsezZjkdSHGI0vZgO7toSg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,276,1716274800"; d="scan'208";a="57423606" Received: from black.fi.intel.com ([10.237.72.28]) by orviesa010.jf.intel.com with ESMTP; 09 Aug 2024 04:49:07 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 6624DA14; Fri, 09 Aug 2024 14:49:02 +0300 (EEST) From: "Kirill A. Shutemov" To: Andrew Morton , "Borislav Petkov (AMD)" , Mel Gorman , Vlastimil Babka Cc: Tom Lendacky , Mike Rapoport , "Matthew Wilcox (Oracle)" , David Hildenbrand , Johannes Weiner , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv2 6/8] mm: Add a helper to accept page Date: Fri, 9 Aug 2024 14:48:52 +0300 Message-ID: <20240809114854.3745464-7-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240809114854.3745464-1-kirill.shutemov@linux.intel.com> References: <20240809114854.3745464-1-kirill.shutemov@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Accept a given struct page and add it free list. The help is useful for physical memory scanners that want to use free unaccepted memory. Signed-off-by: Kirill A. Shutemov Acked-by: David Hildenbrand --- mm/internal.h | 8 ++++++++ mm/page_alloc.c | 53 +++++++++++++++++++++++++++++++++++-------------- 2 files changed, 46 insertions(+), 15 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 1159b04e76a3..ff47d57733ad 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1431,4 +1431,12 @@ unsigned long move_page_tables(struct vm_area_struct= *vma, unsigned long new_addr, unsigned long len, bool need_rmap_locks, bool for_stack); =20 +#ifdef CONFIG_UNACCEPTED_MEMORY +void accept_page(struct page *page); +#else /* CONFIG_UNACCEPTED_MEMORY */ +static inline void accept_page(struct page *page) +{ +} +#endif /* CONFIG_UNACCEPTED_MEMORY */ + #endif /* __MM_INTERNAL_H */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e7e304d9e739..f0610c691ae5 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6980,11 +6980,46 @@ static bool page_contains_unaccepted(struct page *p= age, unsigned int order) return range_contains_unaccepted_memory(start, PAGE_SIZE << order); } =20 +static void __accept_page(struct zone *zone, unsigned long *flags, + struct page *page) +{ + bool last; + + list_del(&page->lru); + last =3D list_empty(&zone->unaccepted_pages); + + account_freepages(zone, -MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE); + __mod_zone_page_state(zone, NR_UNACCEPTED, -MAX_ORDER_NR_PAGES); + __ClearPageUnaccepted(page); + spin_unlock_irqrestore(&zone->lock, *flags); + + accept_memory(page_to_phys(page), PAGE_SIZE << MAX_PAGE_ORDER); + + __free_pages_ok(page, MAX_PAGE_ORDER, FPI_TO_TAIL); + + if (last) + static_branch_dec(&zones_with_unaccepted_pages); +} + +void accept_page(struct page *page) +{ + struct zone *zone =3D page_zone(page); + unsigned long flags; + + spin_lock_irqsave(&zone->lock, flags); + if (!PageUnaccepted(page)) { + spin_unlock_irqrestore(&zone->lock, flags); + return; + } + + /* Unlocks zone->lock */ + __accept_page(zone, &flags, page); +} + static bool try_to_accept_memory_one(struct zone *zone) { unsigned long flags; struct page *page; - bool last; =20 spin_lock_irqsave(&zone->lock, flags); page =3D list_first_entry_or_null(&zone->unaccepted_pages, @@ -6994,20 +7029,8 @@ static bool try_to_accept_memory_one(struct zone *zo= ne) return false; } =20 - list_del(&page->lru); - last =3D list_empty(&zone->unaccepted_pages); - - account_freepages(zone, -MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE); - __mod_zone_page_state(zone, NR_UNACCEPTED, -MAX_ORDER_NR_PAGES); - __ClearPageUnaccepted(page); - spin_unlock_irqrestore(&zone->lock, flags); - - accept_memory(page_to_phys(page), PAGE_SIZE << MAX_PAGE_ORDER); - - __free_pages_ok(page, MAX_PAGE_ORDER, FPI_TO_TAIL); - - if (last) - static_branch_dec(&zones_with_unaccepted_pages); + /* Unlocks zone->lock */ + __accept_page(zone, &flags, page); =20 return true; } --=20 2.43.0 From nobody Fri Dec 19 20:54:07 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2FEB3192B9F for ; Fri, 9 Aug 2024 11:49:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723204157; cv=none; b=pAz5Gn4Wf9iIGPrG8+XdUOI5kL6RHCIcdTw0IpZ+ljTdwzablKRYyukk0ecmHTP4Vdl17za3NLammkdlsrw/igjo/Nqw19bn+9E27XWtxCMV0mxpqB388urdPaBJyHeSKnUuXlMUxYdzxKatUIVJgD58V6615WiEF1WBZ1dbNxk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723204157; c=relaxed/simple; bh=ILver/yjQFclHBCvNLc3z4GjK/nbM+kDF5WFxwHYofI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RbHbpfcZzD/Weil/bm5XhX7DdOBgLyVDXkSNT793bvrakb32y9yrnsVT07VAegaI2Syc4ZUKpeb1jeveMaCa7EzMsOvNhLgl7sdpcCLsPpM2imreKpi0R9GCXtz+Q3o+HEnHayYVSraniary6FND2PXlpZsz48x+OmJt/Zsxp1g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.helo=mgamail.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=g85mnHiB; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.helo=mgamail.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="g85mnHiB" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1723204156; x=1754740156; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ILver/yjQFclHBCvNLc3z4GjK/nbM+kDF5WFxwHYofI=; b=g85mnHiBzs05n7S66jRLg1V/e1s5mjTZVfrf/h7b0ps9PWzJ+e3D88c9 JROqYDOEfYNhxhBDscMPLna0zgbbKmIy//XRxI/TMS7VOEzK/mV7Ta3FC cdm4v3QKRsW5Zpr2IKF67FUd+1gQowDjKp+fNYSf8rF44BNyuwAogAwEv z7PMcnpS0DothF8H/Cr+RIXg4ngVTrECJhbRB1oxQJK9pJBcETAQSDi2L PIDVqErIOZoMHouin1FTchY2VZQXwD3tO5LM3l/hnFpqatt+I8kss2tWE Vlz6eHKUbrz2zbRlRNCbnlkRr/mjcn54VQWPKQjAHx3GtW75hOzvHZOkP w==; X-CSE-ConnectionGUID: mRO+uun/S2qf1OrhWEtWPQ== X-CSE-MsgGUID: T+ztNsdRTCmH74M35Iy65w== X-IronPort-AV: E=McAfee;i="6700,10204,11158"; a="24280359" X-IronPort-AV: E=Sophos;i="6.09,276,1716274800"; d="scan'208";a="24280359" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Aug 2024 04:49:14 -0700 X-CSE-ConnectionGUID: 4fzhaZ4JR7yaZMFTdxS2Bg== X-CSE-MsgGUID: I64npLpmRJC+ZULq1pND6Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,276,1716274800"; d="scan'208";a="57423633" Received: from black.fi.intel.com ([10.237.72.28]) by orviesa010.jf.intel.com with ESMTP; 09 Aug 2024 04:49:10 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 726E7A56; Fri, 09 Aug 2024 14:49:02 +0300 (EEST) From: "Kirill A. Shutemov" To: Andrew Morton , "Borislav Petkov (AMD)" , Mel Gorman , Vlastimil Babka Cc: Tom Lendacky , Mike Rapoport , "Matthew Wilcox (Oracle)" , David Hildenbrand , Johannes Weiner , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv2 7/8] mm: page_isolation: Handle unaccepted memory isolation Date: Fri, 9 Aug 2024 14:48:53 +0300 Message-ID: <20240809114854.3745464-8-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240809114854.3745464-1-kirill.shutemov@linux.intel.com> References: <20240809114854.3745464-1-kirill.shutemov@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Page isolation machinery doesn't know anything about unaccepted memory and considers it non-free. It leads to alloc_contig_pages() failure. Treat unaccepted memory as free and accept memory on pageblock isolation. Once memory is accepted it becomes PageBuddy() and page isolation knows how to deal with them. Signed-off-by: Kirill A. Shutemov --- mm/page_isolation.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/mm/page_isolation.c b/mm/page_isolation.c index 042937d5abe4..39fb8c07aeb7 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -152,6 +152,9 @@ static int set_migratetype_isolate(struct page *page, i= nt migratetype, int isol_ unsigned long flags; unsigned long check_unmovable_start, check_unmovable_end; =20 + if (PageUnaccepted(page)) + accept_page(page); + spin_lock_irqsave(&zone->lock, flags); =20 /* @@ -367,6 +370,11 @@ static int isolate_single_pageblock(unsigned long boun= dary_pfn, int flags, VM_BUG_ON(!page); pfn =3D page_to_pfn(page); =20 + if (PageUnaccepted(page)) { + pfn +=3D MAX_ORDER_NR_PAGES; + continue; + } + if (PageBuddy(page)) { int order =3D buddy_order(page); =20 --=20 2.43.0 From nobody Fri Dec 19 20:54:07 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E9EE81946B8 for ; Fri, 9 Aug 2024 11:49:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723204156; cv=none; b=H6IQ/gWHjZRYfB4x4bdlX7Px0DiGXXi2hYew6isUsmGYrg4SFDOqx57Qp4ygOrzJZxey+QPB+K/jdm6F414vo490p0vuyTsVp4GkuBlippMOlQNRq56OxPQV/TrWWz89JAJ9vS06X1Do1arLE0ETQM/2tegy+A0OAL+uHgQGWQg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723204156; c=relaxed/simple; bh=UJtU8t95QJzWHwTNUBeTKKfK3Fj3/Vd+0O4Q8AZvnwM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ssFRtCc7b1JBVkGFJJYZeNsxYH5zJyXgkuAZL8c0YK5wFGsgA0+wuyyXhlmtMLyXBtfazRJR5EyfMWKAUQxLxBP6WN4npLVGEEkCpI6mSwk5rs7en7Hi+3e9J18eyGlEmXalu+DcYUEFLmjOeKAV2IKl/nUEdSTxUb4t5bJhdp8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.helo=mgamail.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=B78LbE9p; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.helo=mgamail.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="B78LbE9p" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1723204155; x=1754740155; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=UJtU8t95QJzWHwTNUBeTKKfK3Fj3/Vd+0O4Q8AZvnwM=; b=B78LbE9ptf2xDQK1F8Ze65OSHQH837pi3AGUUOw9BZ9zO6Xy1Gjd/cpc gXCWAdPJhVTZ/EbDJkTQ7peKN0T2UzsU15p98OQF/fTZnHN9Jb28fERHV SXtcHlPHWbLXVD3FLxD8rYyuhk1d9/mPJVeb4S9O+1PD7EmYDbMq0MCrT A9lOqHoJffT5ZeOiIo1ScKyTL4rha711xSMPRaJAU9Fi7imMWPiCB8T2Q npkIBFF3TvxBmpoC5/69VbeZjW25J5hQ7gczMM1zzOdb6ju59glmws0m1 rEdr/o5N1u1qI4PLG+CJ60TBdH80VWaNv0CZVZV1+RaK0/FX7vuxhzG13 A==; X-CSE-ConnectionGUID: 5we5DqPlQluNpBGa9Oevow== X-CSE-MsgGUID: HG9E738MRI6P2olMZ9TLVA== X-IronPort-AV: E=McAfee;i="6700,10204,11158"; a="24280354" X-IronPort-AV: E=Sophos;i="6.09,276,1716274800"; d="scan'208";a="24280354" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Aug 2024 04:49:14 -0700 X-CSE-ConnectionGUID: iyxUVt69T5KrH+8owVHsDA== X-CSE-MsgGUID: 5BwJ2V4/SSyOgMZAXr93tQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,276,1716274800"; d="scan'208";a="57506642" Received: from black.fi.intel.com ([10.237.72.28]) by fmviesa008.fm.intel.com with ESMTP; 09 Aug 2024 04:49:10 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 7A445A6B; Fri, 09 Aug 2024 14:49:02 +0300 (EEST) From: "Kirill A. Shutemov" To: Andrew Morton , "Borislav Petkov (AMD)" , Mel Gorman , Vlastimil Babka Cc: Tom Lendacky , Mike Rapoport , "Matthew Wilcox (Oracle)" , David Hildenbrand , Johannes Weiner , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv2 8/8] mm: Accept to promo watermark Date: Fri, 9 Aug 2024 14:48:54 +0300 Message-ID: <20240809114854.3745464-9-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240809114854.3745464-1-kirill.shutemov@linux.intel.com> References: <20240809114854.3745464-1-kirill.shutemov@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Commit c574bbe91703 ("NUMA balancing: optimize page placement for memory tiering system") introduced a new watermark above "high" -- "promo". Accept memory memory to the highest watermark which is WMARK_PROMO now. Signed-off-by: Kirill A. Shutemov --- mm/page_alloc.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index f0610c691ae5..84a7154fde93 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -7046,8 +7046,8 @@ static bool cond_accept_memory(struct zone *zone, uns= igned int order) if (list_empty(&zone->unaccepted_pages)) return false; =20 - /* How much to accept to get to high watermark? */ - to_accept =3D high_wmark_pages(zone) - + /* How much to accept to get to promo watermark? */ + to_accept =3D promo_wmark_pages(zone) - (zone_page_state(zone, NR_FREE_PAGES) - __zone_watermark_unusable_free(zone, order, 0) - zone_page_state(zone, NR_UNACCEPTED)); --=20 2.43.0