From nobody Sun Feb 8 22:53:50 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 174FD1552E1; Mon, 5 Aug 2024 14:59:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722869994; cv=none; b=TPweh6naicUnRc6lvVH+Dj9Gw5LXDnyozElpCV6+1d56tWR9yKcTg8RhEF5GESHdu9BMXkJL4ym4AC92QCivatjcqE6m9SonyOmJZG/hPiKWXiQMTebaJPoxA9tMmgiYuugYxB8qz1cwy6gjlrxdFgdxuBLrm5gkRBTu5Vk0plM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722869994; c=relaxed/simple; bh=wmXqjcepXOvmD9mxVJqk0suOyn9vecB7qQ8hDHFk0qw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ao7xEqWC+jbLK3tsf148jozRNjjdyT6qBUqaPmhbQtONYRFICHHkCfmhVJaOp2BZmC4fMvGEtK/3hyhHP53dWKPuKO246FxORB6+jr5iPbYUPFCSwRtWct/iHcEr8p7ZgftPQR6svNZe0a3BFq6M5b1F0kVWqJB1FE50qdEMUes= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.helo=mgamail.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Zb1ywRae; arc=none smtp.client-ip=198.175.65.18 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.helo=mgamail.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Zb1ywRae" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1722869993; x=1754405993; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=wmXqjcepXOvmD9mxVJqk0suOyn9vecB7qQ8hDHFk0qw=; b=Zb1ywRae7W17L/rnTEv9Nd3LkU0WrghFb1s4cdY/9AQco1FWXLIWAFcx vD2C+8ToyYjdr+iSL3o6171lJwV260eFK08r33teWsxMfwLAhrTbD0KA2 XAdYGQyRR8ClK8w2H4dRFWwAC3sw8v4mGCCleiWizhSpGubo6BdpPN7lO sP1aG+fxYC1L1f4LRz8ZDOufzRuyLTWMvwfLjr4+EqN3SdtLEwstAaXIu xQP10FsSboF7XUWbIoRiosDztgoeIThMd220LQxl7pXEPDGdgXx9qc5VE 0yBCoPGPQheUrGfh4Kohdo9tuhp0+UNUezcr/7PykNYHRW2nCBHyc6eG6 A==; X-CSE-ConnectionGUID: c22WvdOJRtmEZm+x/mGW0Q== X-CSE-MsgGUID: C6FvVCrbRNm3eai/uQfPaA== X-IronPort-AV: E=McAfee;i="6700,10204,11155"; a="20980942" X-IronPort-AV: E=Sophos;i="6.09,264,1716274800"; d="scan'208";a="20980942" Received: from orviesa004.jf.intel.com ([10.64.159.144]) by orvoesa110.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Aug 2024 07:59:50 -0700 X-CSE-ConnectionGUID: +89Um3x/TxS1tnSa7qsbkw== X-CSE-MsgGUID: Nu7NFvCCROW8lc7o1GLDdQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,264,1716274800"; d="scan'208";a="61157248" Received: from black.fi.intel.com ([10.237.72.28]) by orviesa004.jf.intel.com with ESMTP; 05 Aug 2024 07:59:47 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 626E0122; Mon, 05 Aug 2024 17:59:45 +0300 (EEST) From: "Kirill A. Shutemov" To: Andrew Morton , "Borislav Petkov (AMD)" , Mel Gorman , Vlastimil Babka Cc: Tom Lendacky , Mike Rapoport , "Matthew Wilcox (Oracle)" , David Hildenbrand , Johannes Weiner , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Jianxiong Gao , stable@vger.kernel.org Subject: [PATCH 1/8] mm: Fix endless reclaim on machines with unaccepted memory Date: Mon, 5 Aug 2024 17:59:33 +0300 Message-ID: <20240805145940.2911011-2-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240805145940.2911011-1-kirill.shutemov@linux.intel.com> References: <20240805145940.2911011-1-kirill.shutemov@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Unaccepted memory is considered unusable free memory, which is not counted as free on the zone watermark check. This causes get_page_from_freelist() to accept more memory to hit the high watermark, but it creates problems in the reclaim path. The reclaim path encounters a failed zone watermark check and attempts to reclaim memory. This is usually successful, but if there is little or no reclaimable memory, it can result in endless reclaim with little to no progress. This can occur early in the boot process, just after start of the init process when the only reclaimable memory is the page cache of the init executable and its libraries. Make unaccepted memory free from watermark check point of view. This way unaccepted memory will never be the trigger of memory reclaim. Accept more memory in the get_page_from_freelist() if needed. Signed-off-by: Kirill A. Shutemov Reported-by: Jianxiong Gao Fixes: dcdfdd40fa82 ("mm: Add support for unaccepted memory") Cc: stable@vger.kernel.org # v6.5+ Acked-by: David Hildenbrand --- mm/page_alloc.c | 42 ++++++++++++++++++++---------------------- 1 file changed, 20 insertions(+), 22 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 28f80daf5c04..aa9b1eaa638c 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -287,7 +287,7 @@ EXPORT_SYMBOL(nr_online_nodes); =20 static bool page_contains_unaccepted(struct page *page, unsigned int order= ); static void accept_page(struct page *page, unsigned int order); -static bool try_to_accept_memory(struct zone *zone, unsigned int order); +static bool cond_accept_memory(struct zone *zone, unsigned int order); static inline bool has_unaccepted_memory(void); static bool __free_unaccepted(struct page *page); =20 @@ -3072,9 +3072,6 @@ static inline long __zone_watermark_unusable_free(str= uct zone *z, if (!(alloc_flags & ALLOC_CMA)) unusable_free +=3D zone_page_state(z, NR_FREE_CMA_PAGES); #endif -#ifdef CONFIG_UNACCEPTED_MEMORY - unusable_free +=3D zone_page_state(z, NR_UNACCEPTED); -#endif =20 return unusable_free; } @@ -3368,6 +3365,8 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int o= rder, int alloc_flags, } } =20 + cond_accept_memory(zone, order); + /* * Detect whether the number of free pages is below high * watermark. If so, we will decrease pcp->high and free @@ -3393,10 +3392,8 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int = order, int alloc_flags, gfp_mask)) { int ret; =20 - if (has_unaccepted_memory()) { - if (try_to_accept_memory(zone, order)) - goto try_this_zone; - } + if (cond_accept_memory(zone, order)) + goto try_this_zone; =20 #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT /* @@ -3450,10 +3447,8 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int = order, int alloc_flags, =20 return page; } else { - if (has_unaccepted_memory()) { - if (try_to_accept_memory(zone, order)) - goto try_this_zone; - } + if (cond_accept_memory(zone, order)) + goto try_this_zone; =20 #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT /* Try again if zone has deferred pages */ @@ -6951,9 +6946,6 @@ static bool try_to_accept_memory_one(struct zone *zon= e) struct page *page; bool last; =20 - if (list_empty(&zone->unaccepted_pages)) - return false; - spin_lock_irqsave(&zone->lock, flags); page =3D list_first_entry_or_null(&zone->unaccepted_pages, struct page, lru); @@ -6979,23 +6971,29 @@ static bool try_to_accept_memory_one(struct zone *z= one) return true; } =20 -static bool try_to_accept_memory(struct zone *zone, unsigned int order) +static bool cond_accept_memory(struct zone *zone, unsigned int order) { long to_accept; - int ret =3D false; + bool ret =3D false; + + if (!has_unaccepted_memory()) + return false; + + if (list_empty(&zone->unaccepted_pages)) + return false; =20 /* How much to accept to get to high watermark? */ to_accept =3D high_wmark_pages(zone) - (zone_page_state(zone, NR_FREE_PAGES) - - __zone_watermark_unusable_free(zone, order, 0)); + __zone_watermark_unusable_free(zone, order, 0) - + zone_page_state(zone, NR_UNACCEPTED)); =20 - /* Accept at least one page */ - do { + while (to_accept > 0) { if (!try_to_accept_memory_one(zone)) break; ret =3D true; to_accept -=3D MAX_ORDER_NR_PAGES; - } while (to_accept > 0); + } =20 return ret; } @@ -7038,7 +7036,7 @@ static void accept_page(struct page *page, unsigned i= nt order) { } =20 -static bool try_to_accept_memory(struct zone *zone, unsigned int order) +static bool cond_accept_memory(struct zone *zone, unsigned int order) { return false; } --=20 2.43.0 From nobody Sun Feb 8 22:53:50 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 08EB113C827 for ; Mon, 5 Aug 2024 14:59:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722869991; cv=none; b=WPm9abMuzDz6xNxnwF9qaZmgrbdCsx5u5kJuPqtA/QRMoNul0ZRMDVJYaqhOiKLXk24xAxuidzqyHfRHlTscgJalA4fB3bEd5TumoIuGiU49nQz2pjZr0hMhdWOQ0pCTPzekQswnbnQ77gthnIPAsyANwGFoeEyMfcpcMcA15XQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722869991; c=relaxed/simple; bh=mNrd+G8W1pLswdroNW7rKEomKlvtnXxSHo1yiF8wIFI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nnIkBK9xenrEW/kTkc5NGge0Q3QG1FVhbMXc2sP0zUFthg0d7S4xcacbel+L0VsqFz2JDOJxFyBOTXIDWEkduXNcADPdV1mkQgFMLwBJO1eiwyaeNA9dvlmLM9NmtcxqezCycqzGjhxcrWq3ZZ6AEBaCBBeX+/0k+MpbXm43gnI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.helo=mgamail.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=LNeBArFs; arc=none smtp.client-ip=198.175.65.18 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.helo=mgamail.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="LNeBArFs" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1722869990; x=1754405990; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=mNrd+G8W1pLswdroNW7rKEomKlvtnXxSHo1yiF8wIFI=; b=LNeBArFs3daz6PogQTMsu6iecB7YIs8xfE/JcSC8kGoZHvFVJTwRGbD5 pGUwbf6bgrAgJbBsBYYYTxT7dkAmkiHfzSo041zErCp5sZdWpSilNgp1t 7q+bbjoqGGqbj/oTm1Uv+6arAb/UwgVMwEUHHV8i5RibBmmA1YrjeZWXK 9swKjvJw0SiSdlWarqk3PhQ6ccskMo6mfh5ZKCHLl32MM/B8EGd9BKCJ6 d+RyA3l5sBJGWiX0oN5vsh28AXqoAMJKj53cfW/GJ8brOo2gfmWIxTXSz 4TvoGFsqyjXmVnTM0eaTZ5+mS1sXfU3UHf3wGyzzL6N6Gy81Vt7IgQOR0 A==; X-CSE-ConnectionGUID: W8FW3NtXQ+2ScHhe9zbnEg== X-CSE-MsgGUID: RJHFouOrTYqPT5olWhFxiA== X-IronPort-AV: E=McAfee;i="6700,10204,11155"; a="20980909" X-IronPort-AV: E=Sophos;i="6.09,264,1716274800"; d="scan'208";a="20980909" Received: from orviesa004.jf.intel.com ([10.64.159.144]) by orvoesa110.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Aug 2024 07:59:50 -0700 X-CSE-ConnectionGUID: nFlThEekQDSKA6Qsw8j86w== X-CSE-MsgGUID: BYHwffoCSZuA75hvGsy6Og== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,264,1716274800"; d="scan'208";a="61157245" Received: from black.fi.intel.com ([10.237.72.28]) by orviesa004.jf.intel.com with ESMTP; 05 Aug 2024 07:59:46 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 6F7003A8; Mon, 05 Aug 2024 17:59:45 +0300 (EEST) From: "Kirill A. Shutemov" To: Andrew Morton , "Borislav Petkov (AMD)" , Mel Gorman , Vlastimil Babka Cc: Tom Lendacky , Mike Rapoport , "Matthew Wilcox (Oracle)" , David Hildenbrand , Johannes Weiner , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCH 2/8] mm: Accept memory in __alloc_pages_bulk(). Date: Mon, 5 Aug 2024 17:59:34 +0300 Message-ID: <20240805145940.2911011-3-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240805145940.2911011-1-kirill.shutemov@linux.intel.com> References: <20240805145940.2911011-1-kirill.shutemov@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, the kernel only accepts memory in get_page_from_freelist(), but there is another path that directly takes pages from free lists - __alloc_page_bulk(). This function can consume all accepted memory and will resort to __alloc_pages_noprof() if necessary. Conditionally accepted in __alloc_pages_bulk(). The same issue may arise due to deferred page initialization. Kick the deferred initialization machinery before abandoning the zone, as the kernel does in get_page_from_freelist(). Signed-off-by: Kirill A. Shutemov Acked-by: David Hildenbrand Acked-by: Mike Rapoport (Microsoft) --- mm/page_alloc.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index aa9b1eaa638c..90a1f01d5996 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4576,12 +4576,25 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, in= t preferred_nid, goto failed; } =20 + cond_accept_memory(zone, 0); +retry_this_zone: mark =3D wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK) + nr_pages; if (zone_watermark_fast(zone, 0, mark, zonelist_zone_idx(ac.preferred_zoneref), alloc_flags, gfp)) { break; } + + if (cond_accept_memory(zone, 0)) + goto retry_this_zone; + +#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT + /* Try again if zone has deferred pages */ + if (deferred_pages_enabled()) { + if (_deferred_grow_zone(zone, 0)) + goto retry_this_zone; + } +#endif } =20 /* --=20 2.43.0 From nobody Sun Feb 8 22:53:50 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8EFB4142904 for ; Mon, 5 Aug 2024 14:59:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722869993; cv=none; b=PdfEDpMDF2LQMQokM7koPkp3WEgy6RXAeYpCVisuGWTL4uhT07SxoZRVAND2YR+VB4txQMmUEepvuaNh16Qyd2TRsUg6az5gAiZ1cD/OTnZJD2mixV7QBdSU0GTKbTy7WKUtXUTJzbQl6k2TDo9yTcW+NdXzZtB09aFEAxgSBho= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722869993; c=relaxed/simple; bh=Ni1u68tcCrDpDFX2jJrWgh2lUXexRhaqNdEAlro1U9U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nvD77q6wPV31JMXDwH/bOWL6hBP+UWx+TE5Wvo4mXoDZrVTqiKIRC5T2+UQsYk8GUc6IqZ1BiVivoDpKFKlfchghvQkQgG86N5PYw/cfmko2XEAiCz1S9ZEXsOW5SOAROz2cFt7UEou2QyX5X6VEmQzuFf3oUeGQl53sU9BlNRg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.helo=mgamail.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=VL1Rkwrf; arc=none smtp.client-ip=198.175.65.18 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.helo=mgamail.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="VL1Rkwrf" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1722869992; x=1754405992; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Ni1u68tcCrDpDFX2jJrWgh2lUXexRhaqNdEAlro1U9U=; b=VL1Rkwrf30xqLdIlY6IAT5eOELR5K/9t2SeOm6eGBxyZTp93KAx+uOrm kofeLbtDNncbYVbLRHhw6oxVPsGz51dKugFu6J+oL0gmuCQ+R1mi/dCO5 v1gkj2DZsxtH03k8twAxA6T4w0om6A+Mn352BchxsTN3Cy2x2lPqXdKTK nwv17y2FuT1CCnhFQSA+gPp+UWmbZ74x9ckdTXjY+PE4wR00GCSLJ3PPd 69MdddA4HJHZSgSSYnJA4Fas8AX/j1PI2rDqITl+Kbqrm38gHryIHfNV4 eauyoQjAfSYa9TvlMDoH4JzSwlvF0EJzAA+iboeUr68bBOlbT3DUZWzGy g==; X-CSE-ConnectionGUID: sZYrWFt3Q5e0fWN9E19YFA== X-CSE-MsgGUID: kRKZcUUaRBu+9zK9i1kP3A== X-IronPort-AV: E=McAfee;i="6700,10204,11155"; a="20980920" X-IronPort-AV: E=Sophos;i="6.09,264,1716274800"; d="scan'208";a="20980920" Received: from orviesa004.jf.intel.com ([10.64.159.144]) by orvoesa110.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Aug 2024 07:59:50 -0700 X-CSE-ConnectionGUID: EpZlzwukQHO/70HcLBnnoA== X-CSE-MsgGUID: KEjXMtixTu6LFknu5rMdAQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,264,1716274800"; d="scan'208";a="61157246" Received: from black.fi.intel.com ([10.237.72.28]) by orviesa004.jf.intel.com with ESMTP; 05 Aug 2024 07:59:47 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 84FF257E; Mon, 05 Aug 2024 17:59:45 +0300 (EEST) From: "Kirill A. Shutemov" To: Andrew Morton , "Borislav Petkov (AMD)" , Mel Gorman , Vlastimil Babka Cc: Tom Lendacky , Mike Rapoport , "Matthew Wilcox (Oracle)" , David Hildenbrand , Johannes Weiner , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCH 3/8] mm: Introduce PageUnaccepted() page type Date: Mon, 5 Aug 2024 17:59:35 +0300 Message-ID: <20240805145940.2911011-4-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240805145940.2911011-1-kirill.shutemov@linux.intel.com> References: <20240805145940.2911011-1-kirill.shutemov@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The new page type allows physical memory scanners to detect unaccepted memory and handle it accordingly. The page type is serialized with zone lock. Signed-off-by: Kirill A. Shutemov Acked-by: David Hildenbrand --- include/linux/page-flags.h | 3 +++ mm/page_alloc.c | 2 ++ 2 files changed, 5 insertions(+) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 5769fe6e4950..e19eac9c2b5c 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -943,6 +943,7 @@ enum pagetype { PG_hugetlb =3D 0x04000000, PG_slab =3D 0x02000000, PG_zsmalloc =3D 0x01000000, + PG_unaccepted =3D 0x00800000, =20 PAGE_TYPE_BASE =3D 0x80000000, =20 @@ -1076,6 +1077,8 @@ FOLIO_TEST_FLAG_FALSE(hugetlb) =20 PAGE_TYPE_OPS(Zsmalloc, zsmalloc, zsmalloc) =20 +PAGE_TYPE_OPS(Unaccepted, unaccepted, unaccepted) + /** * PageHuge - Determine if the page belongs to hugetlbfs * @page: The page to test. diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 90a1f01d5996..a35efb114496 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6972,6 +6972,7 @@ static bool try_to_accept_memory_one(struct zone *zon= e) =20 account_freepages(zone, -MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE); __mod_zone_page_state(zone, NR_UNACCEPTED, -MAX_ORDER_NR_PAGES); + __ClearPageUnaccepted(page); spin_unlock_irqrestore(&zone->lock, flags); =20 accept_page(page, MAX_PAGE_ORDER); @@ -7030,6 +7031,7 @@ static bool __free_unaccepted(struct page *page) list_add_tail(&page->lru, &zone->unaccepted_pages); account_freepages(zone, MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE); __mod_zone_page_state(zone, NR_UNACCEPTED, MAX_ORDER_NR_PAGES); + __SetPageUnaccepted(page); spin_unlock_irqrestore(&zone->lock, flags); =20 if (first) --=20 2.43.0 From nobody Sun Feb 8 22:53:50 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7B23113C3F9 for ; Mon, 5 Aug 2024 14:59:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722869992; cv=none; b=Jtib3ARgdYvkPNGyLxb7VGivroUe8E8x+yc2IYLw+0GbHKYJnqQwPwavgjZib5ei9xq5ppstQDEDoFzl1aGR3Az9xLdZoREtyAq8dh+rgmJN+pVziw6HR4BdIoD9ul4r5uH7pjEplwGb3HM/pRdz53yjhwojirYDG9djeaOWdnM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722869992; c=relaxed/simple; bh=CYN1tKCYVanVKiVvnUFXLPkw+yb1l3W/6HydRlI1SRs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EUopkqV6+nuNblqslYun1OjWHNBARQjTkbopNsDlHt//+/ZRK2ehOPBnf2QVISrftv1bl3fSez2bt4r0zddWfu/BaCT9q525+wmFvX5UsDVn2ets0np4/U6ZnC+3zb8wRtzqcsbszAX0NhBABybJzCgT9hvLwSvl3CsZaCzTnEE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.helo=mgamail.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=niePIZ2o; arc=none smtp.client-ip=192.198.163.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.helo=mgamail.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="niePIZ2o" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1722869990; x=1754405990; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=CYN1tKCYVanVKiVvnUFXLPkw+yb1l3W/6HydRlI1SRs=; b=niePIZ2oxWugCZe2+MILJ9NWNCmII25gj5Uqhb/SE+XRUXnwFQd33VqL 6wADJ6i3MUkWGg/fwp17D5/LSivmsql6IJy4osfA7pofDZ1z2xeq9sdXm nDjRiYjPOc945QNZ7DiNXXPVPpo4Pz++z9/YSUr2lclzNaTJ/OrZIhq3W KvqJ7OHp76iA7NLSUbfweHSrjExx0hWYpEAWwUT8nextMyUEqh/AoV+wZ K+uGrOFT7emb+7Je+thQcKlWnHDhu5AZvPWmsftXO8XohtIOdMtsTEdH3 SSowC5ttLTkTPkjpwNZfn/uYXOk3RgQvckPe1FW3RQW5Q3YHQuIKyJaRJ A==; X-CSE-ConnectionGUID: EXCddbagSzeKdTGIOQ3JoA== X-CSE-MsgGUID: oTb2PreCQE6buKcYnIe1vQ== X-IronPort-AV: E=McAfee;i="6700,10204,11155"; a="31472749" X-IronPort-AV: E=Sophos;i="6.09,264,1716274800"; d="scan'208";a="31472749" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Aug 2024 07:59:50 -0700 X-CSE-ConnectionGUID: uU2CtQtHQ4iHzmZmtxQd1w== X-CSE-MsgGUID: hxKVXcQ0SZuydzvDC0L52A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,264,1716274800"; d="scan'208";a="56913231" Received: from black.fi.intel.com ([10.237.72.28]) by orviesa008.jf.intel.com with ESMTP; 05 Aug 2024 07:59:47 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 8F5F6628; Mon, 05 Aug 2024 17:59:45 +0300 (EEST) From: "Kirill A. Shutemov" To: Andrew Morton , "Borislav Petkov (AMD)" , Mel Gorman , Vlastimil Babka Cc: Tom Lendacky , Mike Rapoport , "Matthew Wilcox (Oracle)" , David Hildenbrand , Johannes Weiner , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCH 4/8] mm: Rename accept_page() to accept_page_memory() Date: Mon, 5 Aug 2024 17:59:36 +0300 Message-ID: <20240805145940.2911011-5-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240805145940.2911011-1-kirill.shutemov@linux.intel.com> References: <20240805145940.2911011-1-kirill.shutemov@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Rename the helper. The accept_page() name is going to be used for different function. Signed-off-by: Kirill A. Shutemov --- mm/page_alloc.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a35efb114496..34718852d576 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -286,7 +286,7 @@ EXPORT_SYMBOL(nr_online_nodes); #endif =20 static bool page_contains_unaccepted(struct page *page, unsigned int order= ); -static void accept_page(struct page *page, unsigned int order); +static void accept_page_memory(struct page *page, unsigned int order); static bool cond_accept_memory(struct zone *zone, unsigned int order); static inline bool has_unaccepted_memory(void); static bool __free_unaccepted(struct page *page); @@ -1263,7 +1263,7 @@ void __meminit __free_pages_core(struct page *page, u= nsigned int order, if (order =3D=3D MAX_PAGE_ORDER && __free_unaccepted(page)) return; =20 - accept_page(page, order); + accept_page_memory(page, order); } =20 /* @@ -6946,7 +6946,7 @@ static bool page_contains_unaccepted(struct page *pag= e, unsigned int order) return range_contains_unaccepted_memory(start, end); } =20 -static void accept_page(struct page *page, unsigned int order) +static void accept_page_memory(struct page *page, unsigned int order) { phys_addr_t start =3D page_to_phys(page); =20 @@ -6975,7 +6975,7 @@ static bool try_to_accept_memory_one(struct zone *zon= e) __ClearPageUnaccepted(page); spin_unlock_irqrestore(&zone->lock, flags); =20 - accept_page(page, MAX_PAGE_ORDER); + accept_page_memory(page, MAX_PAGE_ORDER); =20 __free_pages_ok(page, MAX_PAGE_ORDER, FPI_TO_TAIL); =20 @@ -7047,7 +7047,7 @@ static bool page_contains_unaccepted(struct page *pag= e, unsigned int order) return false; } =20 -static void accept_page(struct page *page, unsigned int order) +static void accept_page_memory(struct page *page, unsigned int order) { } =20 --=20 2.43.0 From nobody Sun Feb 8 22:53:50 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 74F2C156F41 for ; Mon, 5 Aug 2024 14:59:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722869994; cv=none; b=Nm5Z6nZzf2UDaG6bKjoVszngLMduZf6uf28Uegp7H10mmidjs6yN7kPY1HdIf1ftRrGJMYb0FuVPrsQxxY8LYoYuIrbVojpwcZEY4JUjzdKvzSP6N5HgDN3fYjwq4ZJHIvtDuLGItww1+bApDe8VZVtwouOW3SzD0nJheexbhyU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722869994; c=relaxed/simple; bh=IxVAKdU9izhThF8isuGr3/jEPcqRjyFdz2CDFuPt25A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ldDdRD7v48GJkuLK6CFht4vzYVJP0PDXNt8kv/Mnn2uKJ+dVsTc/hlGblnhal1D/Pceludy6IrFMyJGbhC4BInMePc7v8IeQ5rP/U8TnQigZAlwZLCA1pHymrVQU6IGcZXDe1N7wB69DoTSndym/lN99TEXFCVYyvBfOvYS/M8M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.helo=mgamail.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=fmhMDzbF; arc=none smtp.client-ip=192.198.163.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.helo=mgamail.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="fmhMDzbF" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1722869993; x=1754405993; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=IxVAKdU9izhThF8isuGr3/jEPcqRjyFdz2CDFuPt25A=; b=fmhMDzbFnL8rHwjUZy0m1r9Hzn4hnY37fWkvQsgxW5E9OA1j3D4JCpP8 lgPOZGWhTrDQAahuK5IFU22DZyjel79IM6Gl3hmgR4ew/a/cMyPW5daV2 86mzfNrF2brdqGUwCm+k3STJNXTVbBPct+FTKOSRj0xMd6O0wSOW90535 gX9nbwTUkUXxIiuohu2TWXb79KN+9Ag+fugSTzknEeSx1H6DuEHCwTs/A PH85Onu1qulMQXiMeqK+opwCJY2ZiZ4giz+XpNUpOG/PA723C3+3QKsAn ftHFnVTMVLCLW04AfWEWCkA8lvvcStzXRR4XzEGVHDJTbLIIGwmr27L0J A==; X-CSE-ConnectionGUID: H0N/gi9hTESrds7iB3OimQ== X-CSE-MsgGUID: dUElt27BRju9n3xNcZ0wpA== X-IronPort-AV: E=McAfee;i="6700,10204,11155"; a="31472758" X-IronPort-AV: E=Sophos;i="6.09,264,1716274800"; d="scan'208";a="31472758" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Aug 2024 07:59:53 -0700 X-CSE-ConnectionGUID: 7+wbMWgwSIyNeiuHNZ+IcQ== X-CSE-MsgGUID: mD8ejDxOS2mcd74mvgMbRg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,264,1716274800"; d="scan'208";a="56913243" Received: from black.fi.intel.com ([10.237.72.28]) by orviesa008.jf.intel.com with ESMTP; 05 Aug 2024 07:59:51 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 9939F640; Mon, 05 Aug 2024 17:59:45 +0300 (EEST) From: "Kirill A. Shutemov" To: Andrew Morton , "Borislav Petkov (AMD)" , Mel Gorman , Vlastimil Babka Cc: Tom Lendacky , Mike Rapoport , "Matthew Wilcox (Oracle)" , David Hildenbrand , Johannes Weiner , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCH 5/8] mm: Add a helper to accept page Date: Mon, 5 Aug 2024 17:59:37 +0300 Message-ID: <20240805145940.2911011-6-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240805145940.2911011-1-kirill.shutemov@linux.intel.com> References: <20240805145940.2911011-1-kirill.shutemov@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Accept a given struct page and add it free list. The help is useful for physical memory scanners that want to use free unaccepted memory. Signed-off-by: Kirill A. Shutemov Acked-by: David Hildenbrand --- mm/internal.h | 8 ++++++++ mm/page_alloc.c | 53 +++++++++++++++++++++++++++++++++++-------------- 2 files changed, 46 insertions(+), 15 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index b4d86436565b..d358535f8a7e 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1578,4 +1578,12 @@ void unlink_file_vma_batch_init(struct unlink_vma_fi= le_batch *); void unlink_file_vma_batch_add(struct unlink_vma_file_batch *, struct vm_a= rea_struct *); void unlink_file_vma_batch_final(struct unlink_vma_file_batch *); =20 +#ifdef CONFIG_UNACCEPTED_MEMORY +void accept_page(struct page *page); +#else /* CONFIG_UNACCEPTED_MEMORY */ +static inline void accept_page(struct page *page) +{ +} +#endif /* CONFIG_UNACCEPTED_MEMORY */ + #endif /* __MM_INTERNAL_H */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 34718852d576..5f33e92b0a55 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6953,11 +6953,46 @@ static void accept_page_memory(struct page *page, u= nsigned int order) accept_memory(start, start + (PAGE_SIZE << order)); } =20 +static void __accept_page(struct zone *zone, unsigned long *flags, + struct page *page) +{ + bool last; + + list_del(&page->lru); + last =3D list_empty(&zone->unaccepted_pages); + + account_freepages(zone, -MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE); + __mod_zone_page_state(zone, NR_UNACCEPTED, -MAX_ORDER_NR_PAGES); + __ClearPageUnaccepted(page); + spin_unlock_irqrestore(&zone->lock, *flags); + + accept_page_memory(page, MAX_PAGE_ORDER); + + __free_pages_ok(page, MAX_PAGE_ORDER, FPI_TO_TAIL); + + if (last) + static_branch_dec(&zones_with_unaccepted_pages); +} + +void accept_page(struct page *page) +{ + struct zone *zone =3D page_zone(page); + unsigned long flags; + + spin_lock_irqsave(&zone->lock, flags); + if (!PageUnaccepted(page)) { + spin_unlock_irqrestore(&zone->lock, flags); + return; + } + + /* Unlocks zone->lock */ + __accept_page(zone, &flags, page); +} + static bool try_to_accept_memory_one(struct zone *zone) { unsigned long flags; struct page *page; - bool last; =20 spin_lock_irqsave(&zone->lock, flags); page =3D list_first_entry_or_null(&zone->unaccepted_pages, @@ -6967,20 +7002,8 @@ static bool try_to_accept_memory_one(struct zone *zo= ne) return false; } =20 - list_del(&page->lru); - last =3D list_empty(&zone->unaccepted_pages); - - account_freepages(zone, -MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE); - __mod_zone_page_state(zone, NR_UNACCEPTED, -MAX_ORDER_NR_PAGES); - __ClearPageUnaccepted(page); - spin_unlock_irqrestore(&zone->lock, flags); - - accept_page_memory(page, MAX_PAGE_ORDER); - - __free_pages_ok(page, MAX_PAGE_ORDER, FPI_TO_TAIL); - - if (last) - static_branch_dec(&zones_with_unaccepted_pages); + /* Unlocks zone->lock */ + __accept_page(zone, &flags, page); =20 return true; } --=20 2.43.0 From nobody Sun Feb 8 22:53:50 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 43FFE15AD9E for ; Mon, 5 Aug 2024 14:59:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722869995; cv=none; b=tgUgUeSDD4Wjp1eB9gzq3XgQAdiUw8RfKVxUPf0gv8P9bCBPVCI0T6fUWrwsZJ7wdBlFJXfUHRvZCLSjqzytCEvLYpahbnPIrtpw2xrAdEYycfjb2Ol9Domr9xfhrfrFw8HNO0pxgTelUJQdCzeKrWJpGcpeB2ZrxxuFu9SqlEg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722869995; c=relaxed/simple; bh=ILver/yjQFclHBCvNLc3z4GjK/nbM+kDF5WFxwHYofI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sNVp5Jot0zB2fKsurHwhcgL0BwjhCEFGfD7tijqEs5DVDDMfYVbcxNLaWA0SDnBnqbQwlakQpjyu/iCXtK8KLKPEcr3epX9ClUzFVNe4QIulQn93kXNLnPrYVuXLiN3lL9hmt8i7r0gVdxid3XOp+ouYrBLNAH8gS1w1Jd6nlJg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.helo=mgamail.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=NTwN5PM9; arc=none smtp.client-ip=192.198.163.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.helo=mgamail.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="NTwN5PM9" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1722869994; x=1754405994; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ILver/yjQFclHBCvNLc3z4GjK/nbM+kDF5WFxwHYofI=; b=NTwN5PM9+wdliWx90aOqamoTZpgVL7K7FIQFOP4stJKsPBqQEy/0Rtj0 0UMZ1BcfRvdP72ufBc0mBebG/FG7Mv3wjcnDOxgfQRZQhkDgenJTYF85M +Dygc35wVQwNUAtg+dLRHEGfZuV5kVxu9H4q0eYAEiWfGQExquq4+VwTP kzul97oCbQ2W96r5Ku3lgmwyQui5FdD4j8Hi2hsTYI4my7K8EXod9+Ua6 O+PGaWiJTYmeDpeA6v6a5qf2Rxiune53wTt75FLHpR7y/JF+lBG6mghs/ bVscMLxSEkeccZ9hGbIoNucZ+Kujs1lL+zMda3Q+ib9StFyo/qaJExrd/ g==; X-CSE-ConnectionGUID: kLiHX0A3Tr+wyKOmqVX0wA== X-CSE-MsgGUID: jq3OrPU3QN2K7oALQ6x3Jw== X-IronPort-AV: E=McAfee;i="6700,10204,11155"; a="31472770" X-IronPort-AV: E=Sophos;i="6.09,264,1716274800"; d="scan'208";a="31472770" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Aug 2024 07:59:53 -0700 X-CSE-ConnectionGUID: 544BcHRcRjq358UvEvXv6g== X-CSE-MsgGUID: xeTDIhPcRNix/6cdDWg9Xg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,264,1716274800"; d="scan'208";a="56913244" Received: from black.fi.intel.com ([10.237.72.28]) by orviesa008.jf.intel.com with ESMTP; 05 Aug 2024 07:59:51 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id A7127865; Mon, 05 Aug 2024 17:59:45 +0300 (EEST) From: "Kirill A. Shutemov" To: Andrew Morton , "Borislav Petkov (AMD)" , Mel Gorman , Vlastimil Babka Cc: Tom Lendacky , Mike Rapoport , "Matthew Wilcox (Oracle)" , David Hildenbrand , Johannes Weiner , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCH 6/8] mm: page_isolation: Handle unaccepted memory isolation Date: Mon, 5 Aug 2024 17:59:38 +0300 Message-ID: <20240805145940.2911011-7-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240805145940.2911011-1-kirill.shutemov@linux.intel.com> References: <20240805145940.2911011-1-kirill.shutemov@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Page isolation machinery doesn't know anything about unaccepted memory and considers it non-free. It leads to alloc_contig_pages() failure. Treat unaccepted memory as free and accept memory on pageblock isolation. Once memory is accepted it becomes PageBuddy() and page isolation knows how to deal with them. Signed-off-by: Kirill A. Shutemov --- mm/page_isolation.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/mm/page_isolation.c b/mm/page_isolation.c index 042937d5abe4..39fb8c07aeb7 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -152,6 +152,9 @@ static int set_migratetype_isolate(struct page *page, i= nt migratetype, int isol_ unsigned long flags; unsigned long check_unmovable_start, check_unmovable_end; =20 + if (PageUnaccepted(page)) + accept_page(page); + spin_lock_irqsave(&zone->lock, flags); =20 /* @@ -367,6 +370,11 @@ static int isolate_single_pageblock(unsigned long boun= dary_pfn, int flags, VM_BUG_ON(!page); pfn =3D page_to_pfn(page); =20 + if (PageUnaccepted(page)) { + pfn +=3D MAX_ORDER_NR_PAGES; + continue; + } + if (PageBuddy(page)) { int order =3D buddy_order(page); =20 --=20 2.43.0 From nobody Sun Feb 8 22:53:50 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A5B1815B107 for ; Mon, 5 Aug 2024 14:59:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722869996; cv=none; b=XNdrOn81/4wS9ryax1oTYn8mo1mCKL3TAUM+Nu2la5O8SJqaA0SFE2uM4LQTSn2gyWvmGDQ3bc9bc9YxmV9adC5mlfUxIFdq5Uk9E8EIQi1BVnToeNFMQz2q39Y0gOvbd0nW5jgkkP7G6nDbaKPPCVX2ToxHggM4geUrJ44yJVA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722869996; c=relaxed/simple; bh=HZQzMMA1b4Met0TT3CuOG0tnltrjZmQkpNZYdJg4+Zg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TokVyIfZtR7NEhxvpsC97151gPtHToW3JJbxgILE9Nsp1zK0Q1WEH0yAj/9TBvWEep+mDXkioynpuaeVc9j8WhVfP5M2NKP2r0DqMXVaB0z6JYotQkDoacXdKXQI5aEv2mIY5l9O3em4Fl+iP+QFlWQ/i8JUxEkvdJcdvStn/Ww= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.helo=mgamail.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=ZDikqa6l; arc=none smtp.client-ip=198.175.65.18 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.helo=mgamail.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="ZDikqa6l" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1722869995; x=1754405995; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=HZQzMMA1b4Met0TT3CuOG0tnltrjZmQkpNZYdJg4+Zg=; b=ZDikqa6lIaHJhhGCFuKPtbzl4X7Nwn/Irnys3kmlIZhbtM7k6kYCVGNr mtgReWJwzO+tkiz5WLCOGlOojunlzj/s4JxRG/YoFSnngKqiXONc9me+i +lYGbE78d8YeRaFKfnPxgIuCokMNtEWcws4afbF86sOASC8FxnuxN3Jpz Y9yFESMTREszf+clMIadOME9qGbxK/WQaAoknDmrwjJYixskvMQq7axlz rDr3dU0dEQ+nRqTsUlmxGzYVfpvs1It3e43y+35IrhMZwk4/d6fwbSl0B vZxIY+F5fyxHUmTWyyjd0eyfuU3aPE+qLs+Z7MY8hrU42ehaLJT/Ubxh6 g==; X-CSE-ConnectionGUID: KH5H5auKTomVydDNRjORRA== X-CSE-MsgGUID: JK9WA2ygTha/4iHNoILpbA== X-IronPort-AV: E=McAfee;i="6700,10204,11155"; a="20980974" X-IronPort-AV: E=Sophos;i="6.09,264,1716274800"; d="scan'208";a="20980974" Received: from orviesa004.jf.intel.com ([10.64.159.144]) by orvoesa110.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Aug 2024 07:59:54 -0700 X-CSE-ConnectionGUID: K0ky/TGeTRa3HniYOVe5rw== X-CSE-MsgGUID: e1DtwlPgSQqL5yDU6fySYw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,264,1716274800"; d="scan'208";a="61157259" Received: from black.fi.intel.com ([10.237.72.28]) by orviesa004.jf.intel.com with ESMTP; 05 Aug 2024 07:59:51 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id B03458AC; Mon, 05 Aug 2024 17:59:45 +0300 (EEST) From: "Kirill A. Shutemov" To: Andrew Morton , "Borislav Petkov (AMD)" , Mel Gorman , Vlastimil Babka Cc: Tom Lendacky , Mike Rapoport , "Matthew Wilcox (Oracle)" , David Hildenbrand , Johannes Weiner , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCH 7/8] mm: Introduce promo_wmark_pages() Date: Mon, 5 Aug 2024 17:59:39 +0300 Message-ID: <20240805145940.2911011-8-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240805145940.2911011-1-kirill.shutemov@linux.intel.com> References: <20240805145940.2911011-1-kirill.shutemov@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add promo_wmark_pages() helper to complement other zone watermark accessors. Signed-off-by: Kirill A. Shutemov --- include/linux/mmzone.h | 1 + kernel/sched/fair.c | 2 +- mm/vmscan.c | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 41458892bc8a..eb4ec70f6d53 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -671,6 +671,7 @@ enum zone_watermarks { #define min_wmark_pages(z) (z->_watermark[WMARK_MIN] + z->watermark_boost) #define low_wmark_pages(z) (z->_watermark[WMARK_LOW] + z->watermark_boost) #define high_wmark_pages(z) (z->_watermark[WMARK_HIGH] + z->watermark_boos= t) +#define promo_wmark_pages(z) (z->_watermark[WMARK_PROMO] + z->watermark_bo= ost) #define wmark_pages(z, i) (z->_watermark[i] + z->watermark_boost) =20 /* diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 9057584ec06d..6b2a032ba0c6 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1742,7 +1742,7 @@ static bool pgdat_free_space_enough(struct pglist_dat= a *pgdat) continue; =20 if (zone_watermark_ok(zone, 0, - wmark_pages(zone, WMARK_PROMO) + enough_wmark, + promo_wmark_pages(zone) + enough_wmark, ZONE_MOVABLE, 0)) return true; } diff --git a/mm/vmscan.c b/mm/vmscan.c index cfa839284b92..ad6473eaec28 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -6664,7 +6664,7 @@ static bool pgdat_balanced(pg_data_t *pgdat, int orde= r, int highest_zoneidx) continue; =20 if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) - mark =3D wmark_pages(zone, WMARK_PROMO); + mark =3D promo_wmark_pages(zone); else mark =3D high_wmark_pages(zone); if (zone_watermark_ok_safe(zone, order, mark, highest_zoneidx)) --=20 2.43.0 From nobody Sun Feb 8 22:53:50 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AAE0E158DCC for ; Mon, 5 Aug 2024 14:59:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722869995; cv=none; b=DSC1EOvR0ekYIalwtjXek2pgWSJ5UQgk3mP0PUyZ45XOpFhiT5aQKnqDZyTCf6FJiYzrZIxfPXRZMJZdW5GT9eDxchCU6JodUxLxF2CX0PD2CZHZCO/FIgRX3Vrl/I+7rCXZ1Y4F8romnqnXZuQ37dZcKm4cROdQ1+6Os80maX4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722869995; c=relaxed/simple; bh=l0KMEXpzdLDtoI2/tzFTr7rANQva0av+agV+dlIXFcU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IVxP36ode2aoU9JLYm3vf29tqSgbP2WvBzrsBwSiTcBCJlaOzh2l+XLZ389wc4A5q+Za10YUKGVgttJK20GcRXFTB0gVqOaCM63pHwOmxoCFtTq7I8ag2bmRjTqw4IdYL6jGEprKP+TqpaiSsR6+G5kAjXpDJC5y8fxrsTSS6xg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.helo=mgamail.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=AEQO/VV3; arc=none smtp.client-ip=198.175.65.18 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.helo=mgamail.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="AEQO/VV3" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1722869994; x=1754405994; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=l0KMEXpzdLDtoI2/tzFTr7rANQva0av+agV+dlIXFcU=; b=AEQO/VV3YhYAeTQOVd+jOawwIZd/OmVbWxEATAB074lL1mC7gWJGaBYU hvAa2hVFh/C6hA1qPBbL49P4IFNYUeTGKJPN2oPYq4qIbEAwbYfhGlN47 WHTG2wobq6Y1okCm+7IlLkxXYts2Ww/0iYuCYqwnNint/VszYNc/Vd5A2 N2qJQwt0tib5Y+DMOnU2M+kK/8xpRGajaUX8/wIYHBdb9VSrwuVURQJg1 xvYeCsnDbrO72jpH84P1aPngXSWIhOv9LNTfS97nU7zTjfdflJtr3WlUa ih9zfLdheZL/OdIQncyZOPvGli3EM2n3vEjXnHGl+e2wEakZ6I8nF1Y63 g==; X-CSE-ConnectionGUID: puNrvoMuTYqcTKfeu9szOg== X-CSE-MsgGUID: XxFsv/AhQ1CsgEqAeVcSvw== X-IronPort-AV: E=McAfee;i="6700,10204,11155"; a="20980963" X-IronPort-AV: E=Sophos;i="6.09,264,1716274800"; d="scan'208";a="20980963" Received: from orviesa004.jf.intel.com ([10.64.159.144]) by orvoesa110.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Aug 2024 07:59:54 -0700 X-CSE-ConnectionGUID: WIo67eKrSgGxalNdG6yvhg== X-CSE-MsgGUID: UGUbRBmzTOWsmfDpnnECFQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,264,1716274800"; d="scan'208";a="61157258" Received: from black.fi.intel.com ([10.237.72.28]) by orviesa004.jf.intel.com with ESMTP; 05 Aug 2024 07:59:51 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id BA8168F3; Mon, 05 Aug 2024 17:59:45 +0300 (EEST) From: "Kirill A. Shutemov" To: Andrew Morton , "Borislav Petkov (AMD)" , Mel Gorman , Vlastimil Babka Cc: Tom Lendacky , Mike Rapoport , "Matthew Wilcox (Oracle)" , David Hildenbrand , Johannes Weiner , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCH 8/8] mm: Accept to promo watermark Date: Mon, 5 Aug 2024 17:59:40 +0300 Message-ID: <20240805145940.2911011-9-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240805145940.2911011-1-kirill.shutemov@linux.intel.com> References: <20240805145940.2911011-1-kirill.shutemov@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Commit c574bbe91703 ("NUMA balancing: optimize page placement for memory tiering system") introduced a new watermark above "high" -- "promo". Accept memory memory to the highest watermark which is WMARK_PROMO now. Signed-off-by: Kirill A. Shutemov --- mm/page_alloc.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 5f33e92b0a55..fcb15d0e7549 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -7019,8 +7019,8 @@ static bool cond_accept_memory(struct zone *zone, uns= igned int order) if (list_empty(&zone->unaccepted_pages)) return false; =20 - /* How much to accept to get to high watermark? */ - to_accept =3D high_wmark_pages(zone) - + /* How much to accept to get to promo watermark? */ + to_accept =3D promo_wmark_pages(zone) - (zone_page_state(zone, NR_FREE_PAGES) - __zone_watermark_unusable_free(zone, order, 0) - zone_page_state(zone, NR_UNACCEPTED)); --=20 2.43.0